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Abstract 


Results  from  technology  field  tests  of  the  Digital  Opacity  Compliance  System  (DOCS) 
confirmed  that,  under  fair  weather  conditions  ( i.e .,  clear  sky  conditions),  the  DOCS  technology 
consistently  met  the  quantitative  performance  standards  for  accuracy  and  reliability  defined  for  a 
successful  demonstration.  In  the  field  application  of  the  DOCS  technology  at  real-world 
industrial  operations,  the  DOCS  accuracy  in  measuring  visual  opacity  was  not  only  comparable 
to  Method-9-certified  human  observers  but  the  DOCS  technology  exhibited  several  important 
practical  advantages  for  DoD  facilities  including  1)  improved  measurement  objectivity  and 
reliability,  2)  lower  deployment  and  maintenance  costs  and  3)  generation  of  a  permanent  digital 
image  of  visible  opacity  that  can  be  easily  referenced  in  challenging  regulatory  enforcement 
actions.  These  findings  further  support  the  developer’s  claim  that  the  DOCS  technology  is  a 
more  reliable  opacity  measurement  method  than  Method-9-certified  human  opacity  readers  for 
all  types  of  stationary  sources  and  under  all  weather  conditions. 

Under  adverse  weather  conditions  ( e.g .,  dark  overcast  skies),  the  accuracy  of  both  the  DOCS 
technology  and  Method-9-certified  human  opacity  readers  in  measuring  visible  opacity  was 
diminished.  However,  under  all  weather  conditions,  the  variability  in  the  DOCS  opacity 
measurements  was  significantly  less  than  Method  9,  which  supports  the  claim  that  the  DOCS 
technology  is  consistently  more  reliable  than  Method  9. 

Economic  analysis  of  the  DOCS  technology  illustrated  that  stateside  and  remotely  located  DoD 
facilities  could  recognize  an  annual  cost  savings  of  $9,01 1.82  and  $15,650.10,  respectively,  per 
pair  of  trained  technology  users.  Given  the  fact  that  DoD  currently  certifies  over  3,400 
individuals  trained  in  Method  9,  DoD-wide  adoption  of  the  DOCS  technology  has  the  potential 
of  saving  the  DoD  in  excess  of  15.3  million  dollars  annually  in  compliance  costs.  With  these 
potential  cost  savings,  the  payback  period  for  investment  in  the  DOCS  technology  is  less  than 
one  year.  Using  a  net  present  value  (NPV)  analysis,  evaluation  of  the  life-cycle  cost  savings 
demonstrated  that  DoD  stateside  and  remotely  located  facilities  could  potentially  save 
$40,1 18.82  and  $69,671.12,  respectively,  per  pair  of  trained  technology  users  at  a  facility. 

These  costs  savings  translate  to  a  potential  aggregate  financial  benefit  to  DoD  of  at  least  $68.2 
million  dollars  over  the  life-cycle  of  the  DOCS  equipment  (assuming  a  useful  life  of  five  years). 

Finally,  the  overarching  factor  that  affects  technology  costs  as  well  as  future  technology 
development  is  regulatory  approval  of  a  digital  camera-based  visible  opacity  measurement 
method.  The  future  of  the  DOCS  technology  and  all  other  digital-camera  based  opacity 
measurement  technologies  is  highly  sensitive  to  the  development  and  promulgation  of  an  EPA- 
approved  test  method  for  digital-camera-based  opacity  verification.  Without  such  regulatory 
approval,  the  opportunity  for  the  DOCS  or  similar  technology  to  be  supported  within  the 
marketplace  is  nonexistent. 
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1. 


Introduction 


1.1  Background 

Most  US  Department  of  Defense  (DoD)  installations/facilities  located  in  the  United  States  are 
subject  to  Title  V  of  the  1990  Clean  Air  Act  Amendments.  While  there  are  a  variety  of  air 
sources  regulated  under  Title  V,  the  most  common  are  those  that  generate  visible  emissions,  e.g., 
power  plants,  emergency  generators,  etc.  [2].  While  some  large  sources  employ  dedicated 
equipment  to  measure  and  record  visible  opacity  within  a  stack  {e.g. ,  continuous  opacity 
monitors  or  COMs),  these  systems  are  expensive  to  purchase  and  maintain.  For  the  majority  of 
regulated  air  sources,  the  primary  method  for  determining  compliance  with  permitted  opacity 
levels  is  the  US  Environmental  Protection  Agency’s  (EPA’s)  Reference  Method  9  (Method  9). 
Method  9  relies  on  trained  human  observers  to  visually  determine  compliance  by  estimating  the 
opacity  of  a  smoke  plume  once  every  15  seconds  for  a  specified  period  (Table  1-1). 


Table  1-1 


EPA  Reference  Method  9  Field  Procedures 


Positioning 


•  Observer  must  stand  at  a  distance  sufficient  to  provide  a  clear  view  of  the  visible 
emissions 

•  Observer  must  have  the  sun  oriented  within  a  140°  sector  to  the  back  of  the  observer 

•  Observer  must  ensure  that  the  line  of  vision  is  perpendicular  to  plume  flow  direction 
Recording 

•  Observer  must  record  name  of  the  facility,  emission  location,  facility  type,  observer’s 
name  and  affiliation,  and  the  date  on  which  observations  are  made. 

•  Observer  must  record  time,  estimated  distances  to  the  emission  location,  approximate 
wind  direction  and  speed,  description  of  sky  conditions  and  plume  background  at  the  time 
of  measurement. 

Observations 


•  Observer  must  make  opacity  observations  at  the  point  of  greatest  visible  opacity  where 
condensed  water  vapor  is  absent. 

•  Observer  must  observe  plume  at  15-second  intervals.  Observer  must  not  look 
continuously  at  the  plume. 

•  Observer  must  record  approximate  distance  from  outlet  to  point  in  plume  where 
observations  were  made. 

•  Observer  must  record  opacity  observations  to  the  nearest  5%  opacity  at  15-second 
intervals  on  an  observational  record  sheet  (at  least  24  observations  must  be  recorded). 

Data  Reduction 

•  Observer  shall  detennine  opacity  as  an  average  of  24  consecutive  observations  recorded 
at  15-second  intervals. 
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The  15 -second  opacity  recordings  are  then  averaged  to  detennine  a  single  opacity  estimate  that  is 
compared  against  the  facility’s  permitted  opacity  level  to  demonstrate  compliance  [2,  3],  If  the 
appropriate  field  procedures  are  followed,  a  Method-9-certified  individual  is  legally  authorized  to 
measure  and  report  the  opacity  compliance  status  of  a  regulated  source  (Appendix  C). 

To  become  legally  certified  as  a  Method  9  visual  opacity  observer,  an  individual  must  complete 
both  classroom  training  and  a  visual  opacity  field  examination  at  an  EPA-approved  smoke  school 
once  every  six  months.  The  field  examination  requires  that  the  Method  9  candidate  estimate  the 
visual  opacity  of  25  white  and  25  black  smoke  plumes  with  an  error  rate  of  no  greater  than  15% 
for  any  individual  opacity  observation  and  an  aggregate  opacity  estimation  error  rate  of  no 
greater  than  7.5%  for  all  fifty  (50)  readings  [4].  While  Method  9  has  an  extensive  history  of 
successful  employment,  its  opacity  estimates  are  inherently  subjective  and  completing  the 
process  for  achieving  certification  can  be  expensive.  In  other  words,  since  Method  9  relies  on 
human  observation,  it  is  vulnerable  to  claims  of  inaccuracy,  bias  and,  in  some  cases,  outright 
fraud. 

The  Digital  Opacity  Compliance  System  (DOCS),  which  is  an  innovative  technology  that 
employs  digital  imaging  technology  for  quantifying  visible  opacity,  has  been  developed  and  field 
tested  as  a  technically  defensible  and  economically  competitive  alternative  to  Method  9.  The 
DOCS  uses  a  commercial-off-the-shelf  (COTS)  digital  camera  to  capture  images  of  visible 
opacity,  which  are  then  downloaded  to  a  standard  personal  computer  and  analyzed  using 
statistical  computer  software.  The  DOCS  technology  has  been  advertised  as  not  only  an  accurate 
and  reliable  alternative  technology  to  Method  9  for  quantifying  opacity,  but  it  has  the  added 
advantage  of  furnishing  the  technology  user  a  permanent,  visual  record  of  the  emissions  [5,6]. 

1,2  Objectives  of  the  Demonstration 

The  objective  of  the  present  demonstration  was  to  evaluate  the  technical  field  perfonnance  of  the 
Digital  Opacity  Compliance  System  (DOCS),  a  technology  that  has  been  proposed  as  a  potential 
cost-effective  alternative  to  Method  9  for  measuring  opacity.  The  DOCS  technology  uses  a 
commercial-off-the-shelf  (COTS)  digital  camera  to  capture  images  of  visible  opacity,  which  are 
then  downloaded  to  a  standard  personal  computer  and  analyzed  using  statistical  computer 
software. 

The  current  field  demonstration  was  conducted  in  two  phases.  Phase  I  focused  on  the  evaluation 
of  the  DOCS  technology  at  three  (3)  EPA-approved  Method  9  smoke  schools  located  in  various 
geographical  areas  including  Ogden,  Utah;  Augusta,  Georgia;  and  Columbus,  Ohio.  The  goal  of 
Phase  I  was  to  determine  the  ability  of  the  DOCS  technology  to  achieve  the  Method  9  accuracy 
performance  standard  under  the  same  set  of  field  test  conditions  under  which  human  observers 
were  evaluated.  Phase  II  of  the  field  demonstration  sought  to  broaden  the  technology  evaluation 
conditions  to  include  comparison  of  the  DOCS  technology  with  Method-9-certified  human 
observers  in  estimating  the  opacity  of  air  emissions  associated  with  regulated  processes  at 
various  DoD  industrial  and  commercial  facilities. 

Field  demonstration  results  confirmed  that  the  DOCS  technology  can  accurately  measure  visible 
opacity  at  a  fraction  of  the  cost  required  to  employ  Method-9-certified  observers.  The  financial 
advantage  (as  reflected  in  annual  cost  savings)  conferred  by  utilizing  the  DOCS  technology  at 
stateside  and  remotely  located  DoD  facilities  amounted  to  $9,01 1.82  and  $15,650.10, 
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respectively,  per  pair  of  trained  technology  users  compared  to  Method  9.  With  these  potential 
cost  savings,  the  payback  period  for  investment  in  the  DOCS  technology  is  less  than  one  year. 
Another  inherent  advantage  of  the  DOCS  technology  confirmed  by  the  field  demonstration 
results  was  the  significantly  improved  forensic  reliability  and  reproducibility  in  capturing  visible 
opacity  with  digital  photographs.  The  variability  in  the  DOCS  technology  measurements  was 
found  to  be  significantly  less  that  Method  9  regardless  of  weather  conditions.  These  findings 
were  not  surprising  since,  unlike  Method  9,  which  depends  solely  on  the  visual  judgment  of 
opacity  by  a  human  observer,  the  DOCS  technology  provides  objective  and  reproducible  data. 
The  digital  photograph,  which  may  be  re-examined  at  any  time,  represents  a  visual  archive  of  the 
emissions  as  it  existed  at  the  time  in  question.  In  fact,  the  over  10,000  digital  photographs 
evaluated  in  the  DOCS  field  demonstration  study  are  currently  stored  and  available  on  compact 
discs  and  can  be  reviewed  by  any  interested  third  party,  if  requested. 

Finally,  with  respect  to  regulatory  standards  attained,  field  demonstration  results  confirmed  that 
the  DOCS  can  measure  visible  opacity  of  regulated  stationary  sources  under  favorable  weather 
conditions  with  an  accuracy  that  is  significantly  greater  than  the  codified  Method  9  accuracy 
standard  of  ±7.5%.  Moreover,  within  the  typical  range  of  permitted  regulatory  opacity  (i.e..  0  to 
40  %  opacity),  the  measurement  accuracy  of  the  DOCS  technology  was  comparable  to  that 
achieved  by  Method-9-certified  human  observers. 


1.3  Regulatory  Drivers 

Compliance  with  a  visual  opacity  standard  is  the  most  common  air  quality  requirement  found  in  a 
facility’s  Title  V  operating  permit  [3].  Although  there  is  limited  use  of  continuous  opacity 
monitors  (COMs)  within  regulated  DoD  facilities,  Method  9  is  by  far  the  most  commonly 
prescribed  method  for  estimating  visual  opacity.  Since  the  ultimate  goal  of  the  current  field 
demonstration  is  to  receive  regulatory  approval  for  the  use  of  the  DOCS  technology  by  regulated 
DoD  facilities,  the  EPA’s  Emission  Measurement  Center  (EMC)  in  Research  Triangle  Park, 

N.C.,  as  well  as  a  number  of  Method  9  experts  from  EPA  regional  offices  including  EPA  Region 
VI  (Texas)  and  EPA  Region  VIII  (Colorado)  were  included  as  partners  in  the  current  study.  In 
addition  to  the  federal  environmental  scientists  and  regulators  who  participated  in  the  planning 
and  implementation  of  the  DOCS  technology  field  demonstration  activities,  air  quality  regulators 
from  the  states  of  Alaska,  Texas  and  Utah  provided  valuable  technical  and  regulatory  insight  into 
the  limitations  of  Method  9. 


1.4  Stakeholder/End-User  Issues 

The  use  of  the  DOCS  technology  will  benefit  all  regulated  DoD  facilities  in  verifying  compliance 
with  permitted  visible  opacity  limits.  The  improved  objectivity  in  quantifying  visible  opacity 
levels  (relative  to  the  use  of  Method  9)  through  the  use  of  digital  imagery  will  provide  DoD 
compliance  personnel  with  a  superior  method  in  documenting  their  facility’s  compliance  status. 
The  digital  photographs  of  opacity,  which  may  be  used  independently  (once  the  technology 
receives  regulatory  approval)  or  in  conjunction  with  Method  9  human  observations  for  regulatory 
reporting  purposes,  may  be  analyzed  by  regulators  to  certify  a  regulatory  facility  as  compliant. 
Finally,  the  environment,  in  general,  will  improve  through  the  use  of  the  DOCS  technology 
because  it  confers  to  industrial  facilities  the  ability  to  rapidly  evaluate  and  adjust  their  process 
operating  conditions  to  reduce  visible  air  emissions. 
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2.  Technology  Description 


2.1  Technology  Development  and  Application 

Prior  to  the  current  field  demonstration,  the  DOCS  technology  was  evaluated  at  both  an  EPA- 
approved  smoke  school  and  at  a  limited  number  of  industrial  sites  [5].  Air  quality  inspectors 
employed  at  Hill  Air  Force  Base,  Utah  (Hill  AFB),  have  been  employing  the  DOCS  technology 
to  monitor  the  visible  opacity  associated  with  regulated  air  sources  since  November  1999. 
Because  of  its  early  and  notable  success  at  Hill  AFB,  the  DOCS  technology  is  identified  by  the 
State  of  Utah  Division  of  Air  Quality  (UDAQ)  as  an  allowable  opacity  measurement  option 
available  within  the  Hill  AFB  Title  V  operating  pennit.  Hill  AFB  compliance  personnel  together 
with  UDAQ  are  convinced  of  the  technical  and  economic  advantages  in  using  the  DOCS 
technology  for  opacity  determination  and  are  fully  committed  to  its  future  use.  The 
chronological  history  of  the  DOCS  technology  development  and  funding  support  are  summarized 
in  Table  2-1. 


Table  2-1 


Early  History  of  DOCS  Development 


•  Feasibility  Study  sponsored  by  SCIENTECH,  Inc.,  which  provided  $10,000  to  Utah  State 
University’s  Space  Dynamics  Faboratory  (SDF),  was  focused  on  evaluating  the  general 
concept  of  employing  digital  imagery  for  visible  opacity  estimation. 

•  DOCS  concept  development  and  demonstration  were  conducted  by  SDF  under  a  grant 
funded  though  the  National  Aeronautical  and  Space  Administration  (NASA)  Stennis 
Research  Center’s  Affiliated  Research  Center  Program.  NASA  funds  (ca.  $80,000)  were 
provided  directly  to  the  SDF  to  support  both  staff  and  graduate  student  involvement 
during  an  eight-month  effort. 

•  Funds  to  support  the  DOCS  prototype  development  and  evaluation  of  opacity  estimation 
algorithms  were  provided  by  the  US  Air  Force  under  the  3600  program. 

•  Tyndall  Air  Force  Base,  Florida,  in  collaboration  with  Hill  Air  Force  Base,  Utah,  began 
preliminary  field  testing  of  DOCS,  which  included  the  purchase  and  application  of 
several  DOCS  systems  and  licenses.  The  cost  of  the  field-testing  demonstration  was 
approximately  $32,500. 

•  SCIENTECH,  Inc.,  invested  approximately  $25,000  in  completing  the  development  of 
the  DOCS  analysis  algorithms  and  finishing  the  DOCS  graphic  user  interface  (GUI)  and 
report  generator. 

•  The  State  of  Utah's  Division  of  Air  Quality  (UDAQ)  purchased  one  DOCS  camera  and 
software  license  for  field  application  in  February  2000  at  a  cost  of  $5,000  and  Hill  AFB 
purchased  four  additional  DOCS  cameras  and  software  licenses  in  addition  to  their 
original  site  license  for  $8,000. 
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The  use  of  digital  image  processing  represents  a  unique  and  innovative  approach  for  certifying 
regulatory  compliance  with  visual  opacity  standards.  Other  than  the  Method-9-certified  human 
observer,  the  only  competing  technology  in  use  today  requires  air  emission  sampling  using 
permanently  installed  continuous  opacity  monitors  (COMs).  COMs  have  no  portability  and  are 
costly  both  to  purchase  and  maintain.  Digital  image  processing,  on  the  other  hand,  is  completely 
portable  and  the  analysis  can  be  done  either  on  site,  using  a  standard  notebook  computer,  or  later 
in  the  office,  using  a  common  desktop  system.  Figure  2-1  depicts  how  the  DOCS  technology 
conceptually  manages  digital  imagery  data  to  estimate  the  opacity  associated  with  fugitive  dusts 
as  well  as  the  opacity  associated  with  emissions  from  stationary  sources.  In  all  cases,  the  digital 
images  can  be  archived  by  storing  them  on  computer  media  and  opacity  analysis  can  be 
conducted  at  any  time,  an  advantage  obviously  denied  to  the  Method-9-ccrtificd  human  observer. 


Photographs  of 
Visible  Opacity 
Taken  by 
Commercially 
Available  Digital 
Camera 


■ft  : 

Commercial 
Computer  with  DOCS 
Software  Determines 
Opacity  and  Displays 
Results 


Stationary'  Source 
Opacity  Results 
Displayed 


Figure  2-1 


Schematic  of  DOCS  Technology 


In  the  field  application  of  the  DOCS  technology  to  measure  opacity  associated  with  stationary 
sources,  digital  photographs  of  visible  emissions  are  taken  from  valid  positions  according  to 
codified  Method  9  specifications.  Once  downloaded  to  a  computer  on  which  the  DOCS 
computer  software  has  been  installed,  the  digital  images  can  be  evaluated  for  visible  opacity. 

The  process  for  analyzing  the  digital  image  for  opacity  include  the  following  steps:  1)  activating 
the  DOCS  opacity  computer  program,  2)  retrieving  those  digital  photographs  that  are  to  be 
evaluated  for  visible  opacity  and  3)  using  the  computer  program  to  draw  an  analysis  box  (or  grid) 
around  that  portion  of  the  visible  emissions  that  will  be  analyzed  (Figure  2-2). 

After  the  computer  software  selects  the  purity  of  color  ( i.e saturation)  that  best  corresponds  to 
the  background,  the  opacity  of  the  image  is  calculated  based  on  the  optical  properties  of  the 
pixels  contained  in  the  analysis  box.  The  size  and  shape  of  the  analysis  box,  which  is  controlled 
by  the  user  of  the  software,  must  be  chosen  judiciously  since  the  final  opacity  measurement  will 
ultimately  depend  on  what  part  of  the  image  the  DOCS  computer  software  identifies  as 
background. 
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Figure  2-2 


Use  of  the  DOCS  Software  -  Drawing  the  Opacity  Analysis  Box 


2.2  Previous  Testing  of  the  Technology 

Since  1999,  the  DOCS  technology  has  undergone  a  number  of  successful  field  demonstrations  at 
Hill  Air  Force  Base,  Utah.  These  field  demonstrations  have  included  evaluation  of  the  DOCS 
technology  to  quantify  the  opacity  associated  with  both  regulated  stationary  air  sources  as  well  as 
fugitive  dust  from  range  maintenance  activities  [5],  Prior  to  the  collection  of  any  field  data  in 
support  of  the  present  DOCS  field  demonstration,  an  expert  Opacity  Science  Advisory  Panel 
consisting  of  experienced  scientists  and  engineers  from  DoD,  EPA  and  academia  was  established 
to  conduct  a  comprehensive  review  of  the  science  that  supported  the  DOCS  computer  program 
development  (Table  2-2).  Panel  members  were  selected  based  on  their  demonstrated  expertise  in 
digital  computer-based  algorithm  processing,  measurement  of  opacity  and  smokes,  visibility 
determination  and  modeling,  or  statistical  experimental  design.  After  reporting  that  the 
fundamental  principles  that  supported  the  DOCS  technology  were  scientifically  defensible,  the 
panel  provided  additional  review  and  comment  on  the  DOCS  technology  field  demonstration 
protocol,  statistical  analysis  methods,  demonstration  study  conclusions  and  recommendations. 

2.3  Factors  Affecting  Cost  and  Performance 

The  overarching  factor  that  affects  technology  costs  is  regulatory  approval  of  the  digital  camera- 
based  visible  opacity  measurement  method.  The  future  of  the  DOCS  technology  and  all  other 
digital-camera-based  opacity  measurement  technologies  is  highly  sensitive  to  the  development 
and  promulgation  of  an  EPA-approved  digital-camera-based  opacity-verification  test  method. 
Without  such  regulatory  approval,  the  opportunity  for  the  DOCS  or  similar  technology  to  reach 
the  marketplace  is  nonexistent. 

Beyond  the  financial  impact  stemming  from  regulatory  approval  of  the  DOCS,  better-quality  and 
lower-cost  digital  equipment  is  continuously  being  introduced  into  the  marketplace.  The 
employment  of  digital  cameras  equipped  with  higher  resolution,  faster  processing  times  and/or 
enhanced  zoom  capability  will  allow  the  DOCS  technology  to  be  applied  to  the  measurement  of 
visible  opacity  associated  with  mobile  sources  as  well  as  regulated  sources  located  considerable 
distances  from  the  observer  ( e.g .,  marine  vessels). 
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Another  factor  that  will  impact  cost  and  perfonnance  is  the  enhanced  efficiency  of  the  DOCS 
technology  training  program.  Unlike  Method  9,  DOCS  technology  training  can  be  completely 
computerized  and  made  available  over  the  web  or  through  the  development  and  distribution  of  a 
training  CD.  By  utilizing  standard  information  technology  (IT)  tools,  the  DOCS  technology 
training  program  can  be  designed  to  minimize,  if  not  eliminate,  the  need  to  budget  travel 
expenses  or  hire  temporary  workers  in  support  of  a  DoD  visible  opacity  compliance  program.  It 
is  anticipated  that,  if  the  DOCS  technology  can  be  supported  to  take  full  advantage  of  new 
scientific  and  technical  developments  in  digital  imagery  and/or  information  technology,  the 
DOCS  technology  user  costs  should  continue  to  decrease. 


2.4  Advantages  and  Limitations  of  the  Technology 

Advantages  for  the  DOCS  technology  fall  into  two  distinct  categories.  The  first  category  is 
economic  and,  through  performing  a  simple  cost  analysis,  it  has  become  patently  clear  that  the 
DOCS  technology  has  the  potential  of  saving  the  DoD  considerable  financial  and  personnel 
resources.  The  reduction  in  compliance  costs  associated  with  the  implementation  of  the  DOCS 
technology  stems  primarily  from  the  fact  that  the  type  of  training  necessary  to  support  a  digital 
camera-based  opacity  method  can  be  furnished  through  a  web-based  or  CD-type  program  at  the 
technology  user’s  facility.  A  computer-based  training  program  minimizes  worker  productivity 
loss  as  well  as  eliminates  the  travel  costs  associated  with  supporting  Method  9  smoke  school 
training.  Moreover,  a  web-based  or  CD-type  training  program  can  be  completed  in  a  fraction  of 
the  time  (normally  four  to  eight  hours)  needed  to  support  Method  9  certification  (three  days). 

The  second  category  of  advantages  associated  with  the  DOCS  technology  includes  improved 
forensic  reliability  and  reproducibility.  Method  9  depends  exclusively  on  the  visual  judgment  of 
opacity  [7,  8].  The  visual  observer  records  his  observation  on  a  form  and,  if  needed,  that  form 
and  that  individual  may  be  required  to  provide  expert  testimony  if  legal  enforcement  action  is 
pursued.  The  expert  testimony  then  becomes  merely  an  individual’s  affirmation  that  the 
documented  opacity  is  what  the  observer  recorded  on  that  day.  Because  it  depends  on  human 
judgment,  Method  9  is  inherently  subjective  and,  therefore,  is  vulnerable  to  claims  of  bias, 
inaccuracy  or  outright  fraud.  Alternatively,  the  DOCS  technology  provides  objective, 
reproducible  and  reliable  data.  The  digital  photograph  of  opacity  is  a  visual  archive  of  the 
emissions  as  it  existed  at  the  time  in  question.  The  digital  photograph  may  be  re-examined  at 
any  time  to  establish  a  regulated  source’s  opacity  level. 

Technical  limitations  associated  with  the  DOCS  technology  are  similar  to  those  encountered  by  a 
Method-9-certified  opacity  observer.  Current  regulations  governing  the  measurement  of  visual 
opacity  place  physical  constraints  on  the  angle  of  the  observation  with  respect  to  the  sun  and  the 
distance  to  the  stack  [2,  4,  8].  To  effectively  demonstrate  DOCS  equivalency  to  Method  9,  the 
DOCS  technology  must  be  subjected  to  these  same  constraints.  With  respect  to  background  sky 
conditions,  the  DOCS  technology  field  demonstration  results  confirmed  that  Method-9-certified 
observers  have  significant  difficulty  in  accurately  measuring  the  opacity  of  smoke  plumes  when 
sky  conditions  are  characterized  as  overcast  or  otherwise  highly  variable.  Like  Method-9- 
certified  human  observers,  DOCS  has  similar  limitations  in  that  the  technology  performs  best 
during  clear  blue  sky  days  [6,  8]. 
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3.  Demonstration  Design 


3.1  Performance  Objectives 

The  overarching  objective  of  the  technology  field  demonstration  was  to  evaluate  whether  the 
ability  of  the  DOCS  technology  to  measure  opacity  was  equivalent  to  Method  9.  Because  of  the 
intent  to  petition  federal  regulators  to  approve  the  use  of  the  DOCS  technology  in  lieu  of  Method 
9,  the  EPA’s  Emissions  Measurement  Center  (EMC)  was  invited  to  become  an  active  participant 
in  the  field  demonstration  program.  EMC  involvement  in  the  field  demonstration  activities  was 
integral  in  assessing  both  the  scientific  underpinnings  of  the  DOCS  technology  as  well  as  the 
regulatory  hurdles  associated  with  the  use  of  a  digital-camera-based  opacity  technology  by  the 
regulated  community.  Table  3-1  describes  the  perfonnance  criteria  and  metrics  used  to 
characterize  the  equivalency  of  DOCS  to  Method  9. 


Table  3-1 


Perfonnance  Metrics  Used  to  Establish  DOCS  Performance 


Type  of 
Performance 
Objective 

Primary 

Performance 

Criteria 

Expected  Performance 
(Metric) 

Actual  Performance 
Objective  Met? 

Quantitative 

1 .  Determine 
the  statistical 
equivalency  of 
the  DOCS  to 

EPA  Method  9 

2.  Estimate 
reliability  of  the 
DOCS  relative  to 
EPA  Method  9 
human  observers 

Demonstrate  that  the  DOCS 
opacity  measurements  are 
statistically  within  the 

Method  9  acceptable  margin 
of  error  ( i.e .,  ±  7.5%)  at  the 
99%  confidence  level . 

Compare  the  statistical 
variability  (i.e.,  99% 
confidence  interval)  of  the 
mean  difference  in  opacity 
readings  reported  by  the 
DOCS  to  the  statistical 
variability  (i.e.,  99% 
confidence  interval)  of  the 
mean  opacity  differences 
reported  by  recently 
certified  smoke  readers. 

Yes.  Under  fair  weather 
conditions,  the  DOCS 
technology  was  able  to 
quantify  visible  opacity 
within  the  Method  9 
acceptable  margin  of  error 
at  the  99%  confidence 
level. 

Yes.  The  width  of  the  99% 
confidence  about  the  mean 
opacity  difference  was 
smaller  for  DOCS  than  for 
Method-9-certified  human 
observers  under  all  weather 
conditions  indicating  that 
the  DOCS  is  less  variable 
(i.e.,  more  reliable) 

Qualitative 

1 .  Reproducible 
results 

2.  Near  real¬ 
time  monitoring 

Digital  record 

Increase  observations 

Yes.  Digital  photographs 
of  opacity  were  used  to 
generate  reproducible 
results. 
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3.2  Selecting  Test  Sites/Facilities 

Field  demonstration  sites  were  selected  through  extensive  discussion  between  the  EPA’s 
Emission  Measurement  Center  (EMC)  and  the  Opacity  Science  Advisory  Panel  (Table  2-2). 
During  Phase  I  of  the  field  demonstration  program,  identified  sites  were  selected  based  on  their 
geographic  location  and  the  need  to  include  field  demonstration  sites  whose  weather  patterns 
were  representative  of  the  climates  of  the  Southeast,  Midwest,  and  Western  United  States  (US). 
Following  completion  of  Phase  I,  the  DOCS  technology  was  evaluated  at  various  DoD  industrial 
and  commercial  test  sites  where  there  was  an  acknowledged  and  urgent  need  for  a  digital 
camera-based  opacity  measurement  method  (Phase  II).  Through  discussions  with  the  EPA’s 
Emission  Measurement  Center  (EMC),  EPA  Region  X  and  that  State  of  Alaska  Division  of  Air 
Quality,  the  Phase  II  DOCS  field  demonstration  activities  were  limited  to  those  DoD  industrial 
and  commercial  sites  located  in  the  State  of  Alaska.  Data  collected  during  Phase  II  reflected  the 
performance  of  the  DOCS  technology  under  “real-world’  industrial  conditions  in  an  environment 
that  was  typical  of  Pacific  Northwest  climates. 


3.3  Test  Facility  History/Characteristics 

The  DOCS  Phase  I  field  demonstration  sites  for  the  Method  9  smoke  school  validation  activities 
included  the  cities  of  Ogden,  Utah,  Augusta,  Georgia,  and  Columbus,  Ohio.  The  EPA’s 
Emission  Measurement  Center  recommended  that  the  DOCS  technology  should  be  evaluated  at 
those  EPA-approved  Method  9  smoke  schools  conducted  by  Eastern  Technical  Associates  (ETA, 
Gamer,  North  Carolina).  ETA  currently  conducts  more  than  80%  of  the  smoke  schools  within 
the  United  States  and  there  was  an  established,  professional  working  relationship  between  the 
EPA’s  Emission  Measurement  Center  and  ETA  [8],  The  following  sections  summarize  the  filed 
demonstration  test  sites  in  greater  detail. 


3.3.1  Ogden,  Utah,  Smoke  School 

Hill  Air  Force  Base,  Utah  (Hill  AFB),  is  home  to  the  Ogden  Air  Logistics  Center,  which  is 
located  in  Utah’s  Davis  and  Weber  counties.  Hill  AFB  is  located  at  approximately  4500  feet 
above  sea  level  and  its  normal  weather  patterns  are  typical  of  a  high  mountain  desert  climate. 

Hill  AFB  operates  a  number  of  process  boilers  to  supply  heat  and  process  steam  to  on-base 
facilities.  The  Utah  Division  of  Air  Quality  (UDAQ)  has  established  a  regulatory  limit  for  Hill 
AFB  boilers  of  20%  visible  opacity.  Although  Hill  AFB  utilizes  Method  9  as  its  primary 
opacity  compliance  certification  method,  the  DOCS  system  has  been  employed  at  Hill  AFB  since 
1999  as  an  opacity  monitoring  technology  for  a  range  of  industrial  process  operations,  including 
boilers.  Because  of  its  successful  deployment  at  Hill  AFB,  UDAQ  has  permitted  the  inclusion 
of  the  DOCS  technology  within  Hill  AFB’s  Title  V  operating  permit  as  a  valid  method  with 
which  to  certify  opacity  compliance. 

As  part  of  the  current  DOCS  field  demonstration  activity,  the  DOCS  technology  was  evaluated  at 
an  ETA  sponsored  smoke  school  conducted  at  Weber  State  University  (located  in  Ogden,  Utah) 
from  October  2  through  October  4,  2001.  Weber  State  University  is  located  approximately  6.5 
miles  from  Hill  AFB  and,  like  Hill  AFB,  its  weather  conditions  were  assumed  to  reflect  those 
found  in  high  mountain  desert  climates. 
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3.3.2  Augusta,  Georgia,  Smoke  School 

Fort  Stewart,  Georgia,  is  the  largest  US  Army  installation  east  of  the  Mississippi  River  and  is 
home  to  the  3rd  Mechanized  Infantry  Division.  Early  in  the  site  demonstration  selection  process, 
Fort  Stewart  expressed  an  interest  in  using  the  DOCS  technology  as  a  backup  opacity 
measurement  method  for  its  boiler  stack  continuous  opacity  monitoring  (COM)  system.  To 
provide  the  Fort  Stewart  environmental  personnel  with  DOCS  performance  data  under  climatic 
conditions  similar  to  that  found  at  the  facility,  it  was  decided  to  evaluate  the  DOCS  technology  at 
an  ETA-sponsored  Method  9  smoke  school  located  in  Augusta,  Georgia.  The  smoke  school 
field  demonstration,  which  was  held  October  30,  2001,  through  November  2,  2001,  generated 
DOCS  performance  data  under  climatic  conditions  that  were  assumed  to  be  representative  of 
those  found  at  DoD  facilities  located  throughout  the  Southeast  United  States,  which  would 
include  Fort  Stewart,  Georgia,  among  many  others. 

3.3.3  Columbus,  Ohio,  Smoke  School 

Wright-Patterson  Air  Force  Base,  Ohio  (WPAFB),  is  the  largest,  most  diverse  and 
organizationally  complex  facility  within  the  US  Air  Force.  WPAFB,  which  is  home  to  more 
than  60  different  military  units  representing  a  host  of  Air  Force  and  DoD  organizations,  is 
located  near  Dayton,  Ohio.  WPAFB  operates  two  large  central  heating  plants  that  provide  heat 
and  process  steam  to  many  on-base  facilities.  As  at  Fort  Stewart,  Georgia,  WPAFB’s 
environmental  management  personnel  expressed  a  strong  interest  in  utilizing  the  DOCS  as  a 
backup  opacity  measurement  method  in  support  of  their  COM  units.  To  document  the  DOCS 
technology’s  performance  under  climatic  conditions  representative  of  those  experienced  at 
WPAFB,  the  field  demonstration  sites  included  taking  DOCS  opacity  measurements  at  an  ETA- 
sponsored  Method  9  smoke  school  located  in  Columbus,  Ohio,  from  March  26  through  March 
28,  2002. 

3.3.4  Technology  Demonstration  at  DoD  Industrial  and  Commercial  Facilities 

Phase  II  of  the  field  demonstration  sought  to  broaden  the  technology  evaluation  conditions  to 
include  a  comparison  of  the  DOCS  performance  with  Method-9-certified  human  observers 
estimating  the  opacity  of  air  emissions  associated  with  regulated  processes  at  regulated  industrial 
facilities.  During  the  process  of  identifying  potential  Phase  II  field  demonstration  sites,  EPA’s 
Emission  Measurement  Center  indicated  to  the  DOCS  technology  evaluation  team  that  several 
DoD  and  commercial  facilities  within  the  State  of  Alaska  had  requested  regulatory  approval  for 
the  use  of  the  digital  camera-based  opacity  technology  at  their  facilities.  The  primary  interest  of 
these  facilities  (many  of  which  were  located  in  remote  regions)  in  adopting  a  digital-camera- 
based  opacity-measurement  technology  stemmed  from  the  significant  and  recurring  financial 
costs  associated  with  achieving  and  maintaining  Method  9  certification. 

Through  discussions  among  the  EPA’s  Emission  Measurement  Center,  Opacity  Science 
Advisory  Board,  EPA  Region  X,  State  of  Alaska  Department  of  Environmental  Quality  and 
HMH  Consulting  (Anchorage,  Alaska),  it  was  determined  that  evaluating  the  visible  opacity 
associated  with  the  emissions  from  the  following  regulated  industrial  activities  in  Alaska  would 
represent  a  valuable  test  for  the  DOCS  technology:  1)  waste  incinerator  in  Anchorage,  2)  EPA- 
approved  smoke  generator  in  Anchorage,  3)  coal-fired  power  plant  in  Healy,  4)  coal-fired  boiler 
at  Eielson  AFB,  and  5)  diesel-fired  pump  station  in  Fairbanks.  During  the  first  week  of 
September  2002,  the  visible  emissions  from  each  of  these  field  demonstration  sites  were 
evaluated  using  the  DOCS  technology. 
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3.4  Present  Operations 

At  all  DoD  industrial  and  commercial  facilities  at  which  the  DOCS  technology  was  evaluated, 
Method  9  was  employed  as  either  the  primary  or  secondary  visual  opacity  measurement 
approach.  For  those  industrial  facilities  at  which  Method  9  was  identified  as  a  secondary 
approach  for  measuring  visible  opacity,  the  facility  had  installed  a  continuous  emission  monitor 
(COM)  within  the  air  emission  stack.  Under  normal  conditions,  regulatory  pennits  require  that 
a  back-up  opacity  measurement  system  be  available  when  the  COM  is  malfunctioning  or  is 
otherwise  inoperable. 


3.5  Pre-Demonstration  Testing  and  Analysis 

During  Phase  I  of  the  field  demonstration,  the  DOCS  technology  was  employed  to  estimate  the 
opacity  of  visible  emissions  generated  from  a  15 -foot  stack  operated  by  an  EPA-approved 
Method  9  smoke  school  certification  contractor.  Under  these  test  conditions,  the  ground-truthing 
of  various  black  and  white  smoke  plume  opacities  was  achieved  through  the  use  of  an  in-line 
EPA-certified  transmissometer.  Prior  to  becoming  EPA-certified,  the  transmissometer 
underwent  a  series  of  performance  tests  that  ensured  that  the  instrument  response  was 
consistently  within  a  specified  tolerance  [2,  4,  7,  8],  During  the  evaluation  of  the  Phase  I  field 
demonstration  results,  transmissometer  opacity  readings  were  subsequently  compared  to  those 
recorded  by  the  DOCS  technology. 

Phase  II  of  the  field  demonstration  was  designed  to  evaluate  the  performance  of  the  DOCS 
technology  against  the  ability  of  Method-9-certified  smoke  readers  (/.<?.,  human  observers)  in 
measuring  the  opacity  of  plumes  generated  from  stacks  associated  with  various  industrial 
operations.  It  was  confirmed  before  the  start  of  the  DOCS  technology  field  demonstration 
activities  that  each  individual  human  opacity  reader  had  an  active  Method  9  certification.  To 
qualify  as  a  Method-9-certified  opacity  reader,  an  individual  must  complete  both  classroom 
training  and  a  visual  opacity  field  examination  at  an  EPA-approved  smoke  school  once  every  six 
months.  The  field  examination  requires  that  the  Method  9  candidate  estimate  the  visual  opacity 
of  25  white  and  25  black  smoke  plumes  with  an  error  rate  of  no  greater  than  15%  for  any 
individual  opacity  observation  and  an  aggregate  opacity  estimation  error  rate  of  no  greater  than 
7.5%  for  all  fifty  (50)  readings  [8]. 

3.6  Testing  and  Evaluation  Plan 

3.6.1  Demonstration  Set-Up  and  Start  Up 

The  DOCS  Phase  I  field  demonstration  sites  for  the  Method  9  smoke  school  validation  activities 
included  evaluation  of  the  technology  at  EPA-approved  smoke  schools  conducted  in  the  cities  of 
Ogden,  Utah,  Augusta,  Georgia,  and  Columbus,  Ohio.  The  EPA’s  Emission  Measurement 
Center  (EMC)  recommended  that  the  DOCS  technology  be  tested  at  Method  9  smoke  schools 
managed  by  Eastern  Technical  Associates  (ETA,  Gamer,  North  Carolina).  ETA  currently 
conducts  more  than  80%  of  the  smoke  schools  within  the  United  States  and  its  long-term 
professional  working  relationship  with  EMC  provided  a  high  level  of  confidence  in  the  accuracy 
of  field  results. 

During  Phase  I,  four  (4)  commercial-off-the-shelf  (COTS)  digital  cameras  ( e.g Kodak  DC290 
or  Kodak  DC265)  were  employed  to  photograph  visible  emissions  generated  during  the  EPA- 
approved  Method  9  certification  field  test.  The  DOCS  photographic  imaging  software  was 
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installed  and  tested  on  each  of  the  cameras  before  any  photographs  were  taken.  No  technical 
adjustments  or  physical  modifications  of  the  cameras  were  necessary  to  operate  the  DOCS 
photographic  imaging  software.  The  loading  and  activation  of  the  DOCS  camera  software 
ensured  that  all  of  the  digital  camera’s  optical  settings  were  appropriately  set  to  collect 
photographs  that  could  be  subsequently  analyzed  by  the  DOCS  computer  algorithms. 

Once  the  Kodak  DC290  or  Kodak  DC265  digital  cameras  were  loaded  with  the  DOCS  camera 
software,  each  was  hand  carried  to  the  smoke  school  test  site.  Each  camera  was  positioned  on  a 
tripod  to  provide  a  clear  view  of  the  visible  emissions.  The  minimum  distance  of  the  cameras 
from  the  stack  was  equivalent  to  at  least  three  (3)  stack  heights  (or  45  feet  in  the  present  study) 
with  the  sun  oriented  in  the  140-degree  sector  to  the  back  of  the  camera/observer  (Figure  3-1). 
These  field  procedures  were  adopted  to  be  consistent  with  the  published  requirements  for  valid 
Method  9  visible  emissions  opacity  measurements  [8]. 


Figure  3-1  Positioning  of  the  DOCS  digital  cameras,  Cl,  C2,  C3,  and  C4  and  visual  observers 
during  the  smoke  school  Phase  I  DOCS  field  demonstration. 

Note  that,  as  the  sun  moves  across  the  sky,  both  the  camera  and  visual  observers  must 
shift  their  positions  to  maintain  compliance  with  the  Method  9  opacity  observation 
requirements. 


Phase  II  of  the  field  demonstration  sought  to  broaden  the  technology  evaluation  conditions  to 
include  a  comparison  of  the  DOCS  performance  with  Method-9-certified  human  observers  in 
estimating  the  opacity  of  air  emissions  associated  with  regulated  processes  at  regulated  industrial 
facilities.  A  list  and  description  of  each  of  the  DOCS  technology  Phase  II  field  demonstration 
sites  is  proved  in  Table  3-2. 
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Table  3-2  Description  of  DOCS  Technology  Phase  II  Field  Test  Sites 


1 .  The  Anchorage  Water  and  Wastewater  Sludge  Incinerator:  At  this  field  demonstration  site 
smoke  readings  were  taken  against  a  gray  background  sky.  Smoke  plumes  were  generated 
by  incineration  of  municipal  water  and  wastewater  treatment  sludge. 

2.  The  Golden  Valley  Electric  Power  Station  #1:  This  demonstration  site  provided  an 
opportunity  to  measure  coal-fired  combustor  systems  (black  smoke)  against  a  variable  sky 
background. 

3.  The  Alyeska  Pipeline  Incinerator:  The  Alyeska  pipeline  incinerator  was  barely  able  to 
generate  any  visible  smoke  and  was  unable  to  sustain  any  high  opacity  greater  than  20%  for 
more  than  a  minute  or  two. 

4.  The  Eielson  Coal-Fired  Boiler  Plant:  The  Eielson  coal- fired  boilers  were  also  unable  to 
generate  visible  smoke  for  more  than  very  short  periods  of  time.  The  opacity  that  was 
generated  from  this  facility  was  read  by  both  DOCS  and  certified  human  observers  against  a 
background  characterized  by  overcast  skies. 

5.  The  Williams  North  Pole  Refinery:  Like  the  previous  facilities,  the  opacities  generated  from 
this  facility  were  limited  to  the  low  range  (0  to  20%)  and  were  read  against  a  background 
characterized  by  overcast  grey  skies  and  light  precipitation. 


3.6.2  Period  of  Operation 

The  DOCS  field  demonstration  tests  were  divided  into  two  phases.  Table  3-3  summarizes  the 
dates  and  duration  of  each  phase  of  the  DOCS  technology  field  demonstration. 


3.6.3  Operating  Parameters  for  the  Technology 

During  the  field  study,  four  (4)  commercially  available  digital  cameras  ( e.g .,  Kodak  DC290  or 
Kodak  DC265)  were  employed  to  photograph  visible  emissions  generated  as  part  of  the  EPA 
Reference  Method  9  certification  smoke  school.  The  DOCS  photographic  imaging  software  was 
installed  and  tested  on  each  of  the  cameras  before  any  photographs  were  taken.  Use  of  the 
DOCS  camera  software  is  a  security  requirement  that  essentially  guarantees  that  the  digital 
photograph  cannot  be  altered  prior  to  an  opacity  determination.  No  technical  adjustments  or 
physical  modifications  of  the  cameras  were  necessary  to  operate  the  DOCS  photographic 
imaging  software. 

Each  camera  was  positioned  on  a  tripod  to  provide  a  clear  view  of  the  visible  emissions.  The 
minimum  distance  of  the  cameras  from  the  stack  was  equivalent  to  at  least  three  (3)  stack  heights 
with  the  sun  oriented  in  the  140-degree  sector  to  the  back  of  the  camera/observer.  These  field 
procedures  were  adopted  to  be  consistent  with  the  published  requirements  for  valid  EPA 
Reference  Method  9  visible  emissions  opacity  measurements. 
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Digital  photographs  taken  during  each  day  of  the  DOCS  technology  field  demonstration  were 
collected  on  one  (1)  128-megabyte  (128-MB)  memory  card.  Following  completion  of  each  day 
of  testing,  the  memory  cards  containing  digital  opacity  photographs  were  removed  from  the 
cameras,  placed  in  a  labeled  container,  and  stored  in  a  secure  location.  To  minimize  the  potential 
loss  of  opacity  data,  memory  cards  were  removed  from  the  digital  cameras  only  after  the  digital 
camera’s  power  had  been  turned  off.  A  new  128-MB  memory  card  was  then  inserted  into  each 
camera  for  completion  of  the  following  day’s  field  activities.  Digital  photographs  from  the 
memory  cards  used  during  field  measurements  were  downloaded  daily  to  a  laptop  computer  for 
subsequent  opacity  analysis. 


Table  3-3  Summary  of  Activities  Supporting  the  DOCS  Field  Demonstration 


Phase 

Activity 

Location 

Dates 

Days 

I 

Opacity  Science  Advisory  Panel 
(OSAP)  Meeting 

Ogden,  Utah 

April  2-4,  2001 

3 

I 

OSAP  Meeting 

Ogden 

June  7-9,  2001 

3 

I 

Utah  Smoke  School 

Ogden 

Oct.  2-4,  2001 

4 

I 

OSAP  Meeting 

Washington,  D.C. 

Nov.  7-8,2001 

3 

I 

Manuscript  Report  Summarizing 
Utah  Smoke  School  Results 

Ogden 

Jan.-March  2002 

60 

I 

Georgia  Smoke  School 

Augusta,  Georgia 

Oct.  30-Nov.  1,2001 

4 

I 

Ohio  Smoke  School 

Columbus,  Ohio 

March  26-28,  2002 

4 

I 

OSAP  Meeting 

Ogden 

May  15-17,  2001 

4 

I 

Manuscript  Report  Summarizing  all 
Phase  I  Results 

Ogden 

May-July  2002 

60 

II 

Anchorage  Water  and  Wastewater 
Sludge  Incinerator 

Anchorage,  Alaska 

Sept.  2-7,  2002 

1 

II 

Golden  Valley  Electric  Power 

Station  #1 

Healy,  Alaska 

Sept.  2-7,  2002 

1 

II 

Alyeska  Pipeline  Incinerator 

Healy 

Sept.  2-7,  2002 

1 

II 

Eielson  Coal-Fired  Boiler  Plant 

Fairbanks,  Alaska 

Sept.  2-7,  2002 

1 

II 

Williams  North  Pole  Refinery 

Fairbanks 

Sept.  2-7,  2002 

1 

II 

OSAP  Meeting 

Washington,  D.C. 

Dec.  4-6,  2002 

3 

II 

Manuscript  Report  Summarizing  all 
Phase  II  Results 

Ogden 

Dec.  2002-March 
2003 

60 

I&II 

Initial  Draft  of  Final  Report 

Tyndall  AFB,  Fla. 

May-Oct.  2003 

120 

I&II 

Regulatory  Project  Meeting:  EPA 
EMC;  EPA  Regions  VI  and  VIII; 
states  of  Alaska,  Georgia,  Ohio,  and 
Utah;  DoD  (Air  Force,  Army) 

Research  Triangle 
Park,  N.C. 

June  16-18,  2003 

3 

I&II 

Draft  Cost  &  Performance  Report 

Tyndall  AFB 

Nov.  2003-March 
2004 

120 

I&II 

Revised  Final  Report 

Hill  AFB  (Ogden) 

Dec.  2004-Feb.  2005 

60 

I&II 

Revised  Cost  &  Perfonnance  Report 

Hill  AFB  (Ogden) 

Dec.  2004-Feb.  2005 

60 
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3.6.4  Experimental  Design 

In  both  Phase  I  and  II  of  the  DOCS  field  demonstration,  the  technology  was  limited  to  the 
estimation  of  visible  opacity  associated  with  smoke  plumes  from  stationary  air  sources.  The 
smoke  plumes  were  generated  through  either  an  EPA-approved  Method-9-certification  smoke 
generator  (Phase  I)  or  a  real-world  DoD  industrial  or  commercial  process  operation  (Phase  II). 

Following  completion  of  both  phases  of  the  field  demonstration  data  collection  activities,  the 
opacity  of  each  smoke  plume  captured  as  a  digital  image  was  estimated  using  the  DOCS 
technology  computer  software  by  an  eight-member  panel  consisting  of  federal  government 
civilian  personnel,  US  military  personnel  and  federal  government  contractors.  Each  panel 
member  was  provided  a  compact  disc  that  contained  all  of  the  digital  photographs  taken  from  the 
respective  smoke  school  as  well  as  the  DOCS  computer  software  and  user  guide.  The  panel 
members  were  required  to  work  independently  to  estimate  the  plume  opacity  of  each  digital 
photograph  using  the  furnished  computer  software.  Once  panel  members  had  completed  their 
analyses,  the  opacity  results  were  transferred  and  stored  electronically  in  a  relational  database  for 
subsequent  statistical  evaluation.  An  independent  quality  control  officer  was  assigned  the 
responsibility  of  maintaining  the  integrity  of  all  opacity  data. 

The  following  section  provides  a  brief  summary  of  the  experimental  field  design  including  1)  the 
data  quality  objectives,  2)  brief  description  of  the  EPA-approved  smoke  schools  and  3)  DoD 
industrial  or  commercial  process  operations.  Additional  details  of  the  experimental  design  are 
provided  in  Section  3.3  of  the  current  report. 


3.6.4. 1  Data  Quality  Objectives 

One  of  the  most  effective  approaches  for  developing  a  statistically  defensible  data  collection 
activity  is  the  US  Environmental  Protection  Agency’s  (EPA)  Data  Quality  Objectives  or  DQO 
process  [9].  The  DQO  process  provides  the  environmental  decision-maker  with  a  systematic 
method  for  defining  the  data  quality  criteria  that  a  compliance-sampling  plan  should  satisfy.  The 
data  collection  elements  for  which  specific  data  quality  criteria  are  defined  by  the  DQO  process 
include  the  following:  1)  when  and  where  to  collect  samples  and  2)  the  minimum  number  of 
monitoring  samples  needed  to  support  a  claim  of  regulatory  compliance  with  a  known  level  of 
confidence.  The  current  field  demonstration  project  used  the  EPA’s  DQO  process  for 
evaluating  alternatives  to  Method  9  (Appendix  B). 


3. 6. 4.2  Ogden,  Utah,  Smoke  School 

As  part  of  the  DOCS  technology  field  demonstration  activity,  the  DOCS  technology  was 
evaluated  at  an  ETA-sponsored  smoke  school  conducted  at  Weber  State  University  (located  in 
Ogden,  Utah)  in  October  2001.  Weber  State  University  is  located  approximately  6.5  miles  from 
Hill  AFB  and,  like  Hill  AFB,  its  weather  conditions  were  assumed  to  reflect  those  found  in  high 
mountain  desert  climates.  Additional  details  of  the  Phase  I  experimental  design  are  provided  in 
Section  3.6.1  of  this  report. 


3. 6. 4. 3  Augusta,  Georgia,  Smoke  School 

Fort  Stewart,  Georgia  is  the  largest  US  Army  installation  east  of  the  Mississippi  River  and  is 
home  to  the  3rd  Mechanized  Infantry  Division.  To  provide  the  Fort  Stewart  environmental 
personnel  with  DOCS  technology  performance  data  under  climatic  conditions  similar  to  that 
found  at  the  facility,  it  was  decided  to  evaluate  the  technology  at  an  ETA-sponsored  Method  9 
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smoke  school  located  in  Augusta,  Georgia.  These  field  demonstration  tests  were  conducted 
from  October  30  through  November  2,  200 1 .  Additional  details  of  the  Phase  I  experimental 
design  are  provided  in  Section  3.6.1  of  this  report. 


3. 6. 4.4  Columbus,  Ohio,  Smoke  School 

As  at  Fort  Stewart,  Georgia,  environmental  management  personnel  Wright-Patterson  Air  Force 
Base,  Ohio,  expressed  a  strong  interest  in  utilizing  the  DOCS  technology  as  a  backup  opacity 
measurement  method  in  support  of  their  COM  units.  To  document  the  performance  of  the 
DOCS  technology  under  climatic  conditions  representative  of  those  experienced  at  WPAFB,  the 
field  demonstration  included  application  of  the  DOCS  technology  at  an  ETA-sponsored  Method 
9  smoke  school  located  in  Columbus,  Ohio.  These  field  demonstration  tests  were  conducted 
March  26-28,  2002.  Additional  details  of  the  Phase  I  experimental  design  are  provided  in 
Section  3.6.1  of  this  report. 


3. 6. 4. 5  DoD  Industrial  and  Commercial  Sites 

During  Phase  II,  four  (4)  COTS  digital  cameras  ( e.g .,  Kodak  DC290  or  Kodak  DC265)  were 
employed  to  photograph  visible  emissions  generated  during  the  DOCS  field  demonstration  tests. 
The  DOCS  commercial  digital  imaging  software  was  installed  and  tested  on  each  of  the  cameras 
before  any  photographs  were  taken.  No  technical  adjustments  or  physical  modifications  of  the 
cameras  were  necessary  to  operate  the  DOCS  camera  software. 

Rather  than  assigning  digital  cameras  to  fixed  positions  relative  to  the  sun  (e.g.,  Phase  I),  the 
DOCS  camera  operators  were  allowed  to  position  themselves  in  any  valid  Method  9  location 
relative  to  the  stack  [8].  In  addition  to  the  four  DOCS  camera  operators,  from  six  to  eight 
Method-9-certified  readers  were  available  at  each  DoD  industrial  or  commercial  site  to  estimate 
visible  opacity.  The  Method-9-certified  readers  consisted  of  facility  personnel  as  well  as 
members  of  the  DOCS  technology  evaluation  team. 

During  the  field  test,  the  team  field  coordinator  would  infonn  the  DOCS  camera  operators  and 
Method-9-certified  readers  when  to  begin  evaluating  plume  opacity.  The  camera  operators  and 
Method-9-certified  human  observers  would  then  estimate  plume  opacity  once  every  30  seconds 
for  a  12-minute  period  j  ^Jierating  a  minimum  of  24  readings  and  photographs  per  plume). 

3.6.5  Demobilization 

At  the  conclusion  of  Phase  I  and  Phase  II  DOCS  field  demonstration,  field  equipment  including 
the  Kodak  DC265  or  Kodak  DC  290  digital  cameras  were  powered  off  by  field  technicians 
according  to  the  manufacturer’s  instructions  and  placed  back  in  their  storage  cases.  All 
environmental  monitoring  equipment  were  returned  to  their  storage  cases  by  field  technicians  at 
the  completion  of  the  Phase  I  and  Phase  II  DOCS  field  demonstration.  These  included  the 
following  1)  anemometer,  2)  sling  psychrometer,  3)  Abney  Level  (sun  angle  measurement),  4) 
stop  watch  and  5)  magnetic  compass. 
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3.7  Selection  of  Analytical/Testing  Method 


3.7.1  Weather  Monitoring 

During  both  Phase  I  and  II  DOCS  field  demonstrations,  on-site  field  personnel  recorded  the 
values  of  specific  climatic  parameters  including:  1)  mean  air  temperature,  2)  average  wind  speed, 
3)  maximum  wind  speed,  4)  wind  direction,  5)  sky  conditions,  6)  relative  humidity,  7)  visibility, 
8)  barometric  pressure,  9)  precipitation,  10)  horizontal  sun  angle  and  11)  vertical  sun  angle. 
Methods  used  to  estimate  the  value  of  each  climatic  parameter  are  summarized  in  Table  3-4. 


Table  3-4  Methods  Used  to  Estimate  the  Value  of  Various  Climatic  Parameters 


Parameter 

Method 

Mean  temperature 

Standard  liquid  thermometer  (Eastern  Technical  Associates,  Inc.) 

Average  wind  speed 

Standard  anemometer  (Eastern  Technical  Associates,  Inc.) 

Maximum  wind 
speed 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration  -  www.nws.noaa.gov) 

Wind  direction 

Standard  anemometer  (Eastern  Technical  Associates,  Inc.) 

Sky  conditions 

Visual  observation 

Relative  Humidity 

Sling  Psychrometer  (Eastern  Technical  Associates,  Inc.) 

Visibility 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration  -  www.nws.noaa.gov) 

Barometric  Pressure 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration  -  www.nws.noaa.gov) 

Precipitation 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration  -  www.nws.noaa.gov) 

Horizontal  Sun  Angle 

Magnetic  compass  (Eastern  Technical  Associates,  Inc.) 

Vertical  Sun  Angle 

Abney  level  (Eastern  Technical  Associates,  Inc.) 
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4. 


Performance  Assessment 


4.1  Performance  Criteria 

The  specific  performance  criteria  used  to  evaluate  the  DOCS  technology  perfonnance  are 
summarized  in  Table  4-1.  Prior  to  the  collection  of  any  field  data,  the  characteristics  of  a 
successful  DOCS  field  demonstration  were  defined  to  include  the  following  technology 
performance  criteria:  1)  the  DOCS  technology  should  achieve  the  codified  Method  9  accuracy 
level  at  specified  environmental  conditions  and  2)  the  variability  in  the  DOCS  technology 
opacity  measurements  should  be  demonstrated  to  be  significantly  less  than  what  is  currently 
achieved  by  Method-9-certified  human  smoke  readers.  Achievement  of  these  primary 
technology  performance  criteria  would  be  necessary  to  claim  a  successful  DOCS  field 
demonstration.  Beyond  the  primary  performance  criteria,  establishing  that  the  DOCS  technology 
can  be  operated  without  degradation  in  output  quality  and  documenting  the  level  of  operational 
difficulty  (ease  of  use)  are  important  technology  performance  criteria. 


Table  4-1 


DOCS  Technology  Performance  Criteria 


Performance  Criteria 

Description 

Primary  or 
Secondary 

Validate  as  an  acceptable 
Alternative  to  EPA 

Method  9 

Ensure  the  DOCS  opacity  measurements  are 
statistically  within  the  acceptable  margin  of 
error  established  for  EPA  Method  9 
certification  at  the  99%  confidence  level. 

Primary 

Validate  forensic 
reliability  of  system 

Process  the  DOCS  technology  and  certified 
human  observer  opacity  measurement  data 
and  compute  the  mean  difference  and  the 
associated  99%  confidence  intervals. 

Primary 

Improve  the  amount  of 
quality  data  collected 

Show  the  DOCS  can  operate  continuously 
with  no  degradation  to  the  quality  of  output. 

Secondary 

Ease  of  use 

Show  that  one  trained  individual  can  observe 
multiple  emission  sources  with  one  system. 

Secondary 

4.2  Performance  Confirmation  Methods 

To  establish  whether  the  DOCS  technology  is  capable  of  estimating  visible  emissions  from 
stationary  air  pollutant  sources  as  accurately  and  reliably  as  EPA  Reference  Method  9  visible 
emission  observers,  opacity  measurements  recorded  by  both  methods  were  collected  during  the 
field  demonstration.  During  Phase  I  of  the  field  demonstration,  the  primary  source  of  data 
required  to  evaluate  the  perfonnance  of  the  DOCS  technology  consisted  of  opacity  readings 
recorded  simultaneously  by  1)  an  EPA-certified  in-line  transmissometer,  2)  Method-9-certified 
visible  opacity  human  observers  and  3)  the  DOCS  technology.  Statistical  analysis  of  the  mean 
difference  in  opacity  measurements  generated  by  comparing  the  opacity  readings  recorded  by  the 
EPA  certified  in-line  transmissometer  and  the  opacity  readings  recorded  by  both  the  Method-9- 
certified  human  observers  and  the  DOCS  technology  served  as  the  technical  basis  for  comparing 
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the  equivalency  of  the  two  technologies.  If  the  DOCS  technology  estimated  visible  opacity  with 
an  accuracy  that  was  equal  to  or  better  than  the  codified  Method  9  accuracy  level  ( i.e .,  7.5%)  at 
the  99%  confidence  level,  the  conclusion  would  be  that  one  of  the  primary  performance  criteria 
has  been  achieved.  Similarly,  if  the  width  of  the  99%  confidence  interval  (99%  Cl)  associated 
with  the  DOCS  technology  measurements  were  demonstrated  to  be  significantly  smaller  than  the 
99%  Cl  associated  with  the  Method-9-certified  human  opacity  observations,  the  conclusion 
drawn  would  be  that  both  primary  performance  criteria  have  been  met. 

During  Phase  II  of  the  DOCS  field  demonstration,  not  all  regulated  air  sources  were  equipped 
with  in-line  transmissometers  {i.e.,  continuous  opacity  monitors).  Without  the  benefit  of  a 
reliable  and  regulatory  approved  opacity  monitor,  a  comparison  of  the  accuracy  of  the  DOCS 
relative  to  Method-9-certified  smoke  readers  could  not  be  achieved.  However,  opacity  data 
collected  from  air  sources  not  equipped  with  opacity  sensors  could  be  used  to  determine  whether 
the  DOCS  and  Method-9-certified  opacity  readers  yielded  field  results  that  were  statistically 
different.  Therefore,  the  opacity  data  collected  at  the  DoD  industrial  and  commercial  sites  were 
analyzed  differently,  based  on  whether  the  DOCS  opacity  measurements  could  be  verified  by  a 
continuous  opacity  monitor  (COM).  In  other  words,  for  those  air  sources  equipped  with  COMs, 
the  DOCS  field  demonstration  data  were  managed  in  a  comparable  manor  to  the  data  collected 
during  Phase  I  {i.e.,  conclusions  could  be  drawn  regarding  the  technology’s  accuracy).  For 
those  air  sources  not  equipped  with  COMs,  a  simple  comparison  of  the  technology’s  reliability 
was  conducted. 


4.3  Data  Analysis,  Interpretation  and  Evaluation 

4.3.1  Phase  I — Ogden,  Utah,  Smoke  School 

During  the  three  (3)  days  of  DOCS  field  testing  at  the  Ogden,  Utah,  site,  weather  conditions 
were  near  perfect  for  measuring  visual  opacity  {i.e.,  clear  blue  skies,  low  wind  and  low 
humidity — Table  4-2). 


Table  4-2 


Climatic  Conditions  in  Ogden,  Utah 


Parameter 

Day  of  Test 

Day  1 

Day  2 

Day  3 

Mean  Temperature  (°  F) 

66.2 

60.8 

60.8 

Average  Wind  Speed  (mph) 

8.9 

9.7 

8.1 

Maximum  Wind  Speed  (mph) 

13.8 

16.1 

11.4 

Wind  Direction 

N-NW 

E-SE 

S-SE 

Sky  Conditions 

Clear 

Clear 

Clear 

Relative  Humidity  (percent) 

27.2 

45.2 

30.5 

Visibility  (miles) 

7.0 

7.0 

7.0 

Barometric  Pressure  (in.  of  Hg) 

30.08 

30.11 

30.01 

Precipitation  (in.) 

0 

0 

0 

Vertical  Sun  Angle  (degrees) 

42.2 

39.4 

38.5 
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During  the  Ogden,  Utah,  smoke  school,  approximately  4,741  digital  images  (2,336  black  plumes 
and  2,405  white  plumes)  were  analyzed  for  visible  opacity  using  the  DOCS  computer  software. 
The  decision  to  exclude  some  images  from  the  DOCS  analyses  was  based  on  a  number  of 
technical  problems  including  1)  physical  obstruction  of  the  smoke  plume  ( e.g .,  trees,  clouds, 
telephone  poles,  etc.)  and  2)  folding,  twisting  or  other  significant  physical  disruptions  to  the 
plume. 

4. 3. 1.1  Quantitative  Analysis — Ogden,  Utah,  Smoke  School 

The  statistical  parameters  identified  as  critical  in  the  technical  evaluation  of  DOCS  were  1)  the 
absolute  value  of  the  mean  opacity  difference  {e.g.,  opacity  measurement  recorded  by  DOCS 
minus  the  opacity  measurement  reported  by  the  EPA-approved  transmissometer)  and  2)  99% 
confidence  interval  of  the  mean  opacity  difference.  Table  4-3  summarizes  the  statistical  results 
from  the  Ogden,  Utah,  field  trial  of  the  DOCS. 


Table  4-3  Statistical  Data  Summary  of  Ogden,  Utah,  Smoke  School  DOCS  Evaluation 


Color  of  Smoke — 

Opacity  Measurement  Approacl 

Opacity 

Range 

(%) 

Mean 

Difference 

(%) 

Number  of 
Samples 

99%  Cl1 

Black— DOCSJ 

0-100 

6.4 

2336 

6.0-6. 8 

Black — certified  observers 

0-100 

6.7 

246 

5.4-8. 1 

Black— DOCS 

0-60 

5.6 

1957 

5. 2-6.0 

Black — certified  observers 

0-60 

5.4 

212 

4. 1-6.8 

Black— DOCS 

0-40 

5.4 

1745 

4.9-5. 8 

Black — certified  observers 

0-40 

4.8 

194 

3.5-6. 1 

White— DOCS 

0-100 

10.0 

2405 

9.4-10.6 

White — certified  observers 

0-100 

8.5 

282 

7.1-10.0 

White— DOCS 

0-60 

6.7 

1897 

6.2-12 

White — certified  observers 

0-60 

8.2 

224 

6.6-9. 8 

White— DOCS 

0-40 

5.9 

1686 

5. 4-6.3 

White — certified  observers 

0-40 

7.4 

199 

5. 7-9.0 

'99%  Cl  =  99%  confidence  interval 


Over  the  full  range  of  opacity  (/.£.,  0  to  100%),  the  mean  opacity  difference  (/.<?.,  DOCS  opacity 
reading  minus  the  transmissometer  opacity  reading)  for  black  smoke  was  estimated  to  be  6.4  % 
with  a  99  %  confidence  interval  that  ranged  from  6.0  to  6.8.  These  results  demonstrate  that,  for 
black  smoke,  the  DOCS  has  a  margin  of  error  in  measuring  plume  opacity  that  is  significantly 
less  than  the  acceptable  margin  of  error  associated  with  Method  9  {i.e.,  7.5%)  [8].  These 
statistical  results,  which  specifically  address  the  primary  technology  perfonnance  criteria, 
support  the  conclusion  that  the  accuracy  of  the  DOCS  to  quantify  the  visible  opacity  of  black 
smoke  is  equal  to  or  greater  than  what  is  required  to  certify  under  Method  9. 
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Similarly,  the  field  data  indicated  that,  when  the  opacity  of  white  smoke  was  held  below  60%, 
the  DOCS  technology  consistently  achieved  the  Method  9  accuracy  standard.  As  for  black 
smoke,  these  results  demonstrate  that  the  DOCS  technology  field  test  results  fully  meet  the 
primary  perfonnance  criteria.  In  contrast,  within  the  same  opacity  range,  Method-9-certified 
visual  observers  reported  opacity  measurements  that  yielded  mean  differences  that  were 
significantly  greater  than  the  acceptable  margin  of  error  associated  with  Method  9  [8]. 

With  the  overwhelming  success  of  the  DOCS  performance  at  the  Ogden,  Utah,  smoke  school,  the 
Opacity  Science  Advisory  Panel  in  conjunction  with  the  EPA’s  Emission  Measurement  Center 
requested  that  a  manuscript  summarizing  the  findings  from  the  field  demonstration  be  prepared 
and  submitted  to  a  peer-reviewed  technical  journal.  The  Hill  AFB  environmental  management 
directorate  accepted  the  task  of  drafting  the  technical  manuscript,  which  was  published  by  the 
Journal  of  the  Air  &  Waste  Management  Association  ( JAWMA )  in  2003  [6]. 

4.3.2  Phase  I — Augusta.  Georgia.  Smoke  School 

The  climatic  conditions  recorded  during  the  Augusta,  Georgia,  smoke  school  were  characterized 
by  scattered  clouds  and  partly  overcast  skies  (Table  4-4).  Not  only  were  the  weather  conditions 
in  Augusta,  Georgia,  found  to  be  appreciably  different  from  those  encountered  in  Ogden,  Utah, 
but  the  physical  landscape  of  the  two  smoke  school  locations  were  drastically  different  as  well. 
For  example,  while  the  Ogden,  Utah,  smoke  school  field  tests  were  conducted  in  a  large  open 
parking  lot  located  adjacent  to  a  university  athletic  stadium,  the  venue  for  the  Georgia  smoke 
school  was  a  smaller  and  more  secluded  parking  lot  surrounded  by  a  dense  pine  forest.  The 
combination  of  cloudy  conditions  and  tall  trees  provided  a  more  variable  background  against 
which  the  DOCS  and  Method  9-certified  human  observers  were  estimating  plume  opacity. 


Table  4-4  Climatic  Conditions  During  DOCS  Smoke  Field  Demonstration  in  Augusta, 
Georgia 


Parameter 

Day  of  Test 

Day  1 

Day  2 

Day  3 

Mean  Temperature  (°F) 

63.1 

60.0 

67.5 

Average  Wind  Speed  (mph) 

4.0 

5.3 

5.0 

Maximum  Wind  Speed  (mph) 

4.6 

11.5 

9.2 

Wind  Direction 

N 

N 

S-SE 

Sky  Conditions 

Partly  Cloudy 

Scattered 

Clouds 

Cloudy 

Relative  Humidity  (percent) 

50.9 

72.0 

90.9 

Visibility  (miles) 

10.0 

10.0 

7.9 

Barometric  Pressure  (in.  of 

Hg) 

30.36 

30.42 

30.27 

Precipitation  (in.) 

0 

0 

0 

Vertical  Sun  Angle  (degrees) 

32.4 

31.0 

28.6 
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4.3.2. 1  Quantitative  Analysis — Augusta,  Georgia,  Smoke  School 

Table  4-5  provides  a  summary  of  the  DOCS  technology  statistical  data  generated  from  the 
Augusta,  Georgia,  smoke  school  field  demonstration  tests.  Over  the  full  range  of  opacity  for 
black  smoke,  neither  the  DOCS  technology  nor  the  Method-9-certified  human  opacity  readers 
were  able  to  meet  the  accuracy  requirements  established  by  EPA  Reference  Method  9.  The  mean 
opacity  difference  recorded  by  the  DOCS  technology  for  black  smoke  was  8.6%  while  Method- 
9-certified  readers  recorded  a  mean  difference  of  8.4%.  Despite  the  inability  of  the  DOCS 
technology  to  achieve  the  accuracy  standard  specified  for  supporting  method  equivalency  during 
the  Augusta,  Georgia,  field  tests,  the  variability  in  the  DOCS  analyses  (as  reflected  in  the  range 
of  the  statistical  confidence  intervals)  was  significantly  less  than  that  found  for  Method-9- 
certified  smoke  readers  [6].  This  observation  supports  one  of  the  primary  field  demonstration 
performance  criteria,  namely  that  the  DOCS  technology  is  inherently  more  reliable  than  Method  9. 


Table  4-5  Statistical  Data  Summary  of  Augusta,  Georgia,  Smoke  School  DOCS  Evaluation 


Color  of  Smoke — 

Opacity  Measurement  Approacl 

Opacity 
Range  (%) 

Mean 

Deviation 

(%) 

Number 

of 

Samples 

99%  Cl1 

Black— DOCS 

0-100 

8.6 

4949 

8.3 -9.0 

Black — certified  observers 

0-100 

8.4 

543 

5.5-11.3 

Black— DOCS 

0-60 

8.2 

3620 

7.8 -8.6 

Black — certified  observers 

0-60 

6.1 

398 

5.2  -  7.1 

Black— DOCS 

0-40 

7.9 

2896 

7.4 -8.3 

Black — certified  observers 

0-40 

4.7 

315 

3.8 -5.6 

White— DOCS 

0-100 

13.2 

3535 

12.5-13.9 

White — certified  observers 

0-100 

6.2 

365 

5.2 -7.2 

White— DOCS 

0-60 

8.5 

2565 

8.0  -  9.1 

White — certified  observers 

0-60 

4.9 

265 

4.0 -5.8 

White— DOCS 

0-40 

7.2 

2203 

6.6 -7.7 

White — certified  observers 

0-40 

4.1 

227 

3.4 -4.9 

'99%  Cl  -  99%  confidence  interval 


Over  the  limited  opacity  range  of  0  to  60%  for  black  smoke,  Method-9-certified  readers  appear  to 
be  more  accurate  than  the  DOCS  with  a  reported  mean  difference  of  6. 1%  compared  to  8.2% 
estimated  for  the  DOCS.  The  reasons  for  the  improved  accuracy  of  Method-9-certified  smoke 
readers  to  quantify  opacity  under  the  smoke  school  environment  was  most  likely  due  to  the  fact 
that,  with  the  relatively  small  stack  height  associated  with  the  EPA  smoke  generator  ( ca .  15  feet), 
human  observers  could  easily  utilize  a  background  other  than  sky  ( e.g .,  vegetation,  telephone 
poles,  bill  boards,  etc.)  to  improve  the  level  of  contrast  exhibited  by  the  plume,  while  the  DOCS 
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was  limited  to  the  use  of  sky.  Although  utilization  of  background  other  than  sky  is  permissible 
during  Method  9  smoke  certification  testing,  during  the  regulatory  application  of  Method  9, 
human  observers  often  have  to  estimate  the  opacity  of  plumes  generated  from  stacks  that  are 
positioned  on  top  of  buildings  or  are  otherwise  located  above  the  local  tree  lines.  Under  these 
conditions,  the  visual  observer  is  compelled  to  utilize  sky  as  background. 

4.3.3  Phase  I  Columbus.  Ohio,  Smoke  School 

The  climatic  conditions  recorded  during  the  Columbus,  Ohio,  smoke  school  are  summarized  in 
Table  4-6.  With  respect  to  evaluating  opacity,  the  weather  conditions  during  the  Columbus,  Ohio, 
field  tests  were  considerably  poorer  when  compared  to  either  the  Utah  or  Georgia  smoke  schools. 
The  weather  conditions  in  Columbus,  Ohio,  for  the  DOCS  technology  field  tests  were 
characterized  by  freezing  temperatures,  light  rain  mixed  with  snow,  and  thick,  overcast  skies. 


Table  4-6  Climatic  Conditions  During  DOCS  Smoke  Field  Demonstration  in  Columbus,  Ohio 


Parameter 

Day  of  Test 

Day  1 

Day  2 

Day  3 

Mean  Temperature  (°  F) 

32.0 

33.0 

37.0 

Average  Wind  Speed  (mph) 

7.8 

6.9 

5.1 

Maximum  Wind  Speed  (mph) 

13.8 

11.5 

13.8 

Wind  Direction 

NW 

NW 

S 

Sky  Conditions 

Freezing  Rain, 
Overcast 

Haze,  Overcast 

Overcast, 

Scattered 

Clouds 

Relative  Humidity  (percent) 

93.0 

83.2 

95.0 

Visibility  (miles) 

5.5 

8.5 

5.7 

Barometric  Pressure  (in.  of 

Hg) 

29.98 

30.18 

30.15 

Precipitation  (in.) 

0.77 

0 

0 

Vertical  Sun  Angle  (degrees) 

no  sun 

38.0 

46.0 

4.3.3. 1  Quantitative  Analysis — Columbus,  Ohio,  Smoke  School 

A  summary  of  the  DOCS  technology  statistical  data  obtained  from  the  Columbus,  Ohio,  smoke 
school  is  presented  in  Table  4-7.  Over  the  full  range  of  opacity  for  both  black  and  white  smoke, 
neither  the  DOCS  technology  nor  the  Method-9-certified  human  opacity  readers  were  able  to 
achieve  the  accuracy  standard  defined  by  Method  9.  The  failure  of  both  the  DOCS  and  Method- 
9-certified  smoke  readers  to  achieve  the  Method  9  accuracy  levels  was  unexpected  but  not 
surprising  given  the  challenging  weather  conditions.  A  common  concern  expressed  by  many  of 
the  smoke  school  participants  was  their  inability  to  actually  visualize  the  plume  using  the  sky  as 
background. 
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Table  4-7  Statistical  Data  Summary  of  the  Columbus,  Ohio,  Smoke  School  DOCS  Evaluation 


Color  of  Smoke — 

Opacity  Measurement  Approach 

Opacity 
Range  (%) 

Mean  Deviation 
(%) 

Number  of 
Samples 

99%  Cl1 

Black— DOCS 

0-100 

10.9 

3498 

10.4-11.5 

Black — certified  observers 

0-100 

12.0 

1492 

11.3-12.7 

Black— DOCS 

0-60 

9.4 

3066 

8.9 -9.8 

Black — certified  observers 

0-60 

10.6 

1192 

9.9-11.3 

Black— DOCS 

0-40 

8.1 

2753 

7.6 -8.5 

Black — certified  observers 

0-40 

9.2 

1012 

8.5 -9.9 

White— DOCS 

0-100 

21.6 

4394 

20.8-22.4 

White — certified  observers 

0-100 

10.0 

1500 

9.4-10.6 

White— DOCS 

0-60 

15.0 

3758 

14.4-15.5 

White — certified  observers 

0-60 

9.7 

1176 

9.1  -  10.4 

White— DOCS 

0-40 

12.3 

3131 

11.9-12.7 

White — certified  observers 

0-40 

9.5 

1020 

8.8-10.3 

'99%  Cl  =  99%  confidence  interval 


4.3.4  Significance  Testing 

The  initial  documentation  of  method  equivalency  was  established  through  computing  the 
confidence  intervals  about  the  mean  difference  between  the  DOCS  technology  and  EPA-certified 
transmissometer  opacity  readings.  Based  on  a  recommendation  received  from  the  EPA’s 
Emission  Measurement  Center,  statistical  significance  testing  was  conducted  as  well. 

In  significance  testing,  a  null  hypothesis  (//„)  is  developed  that  will  be  assumed  to  be  true  in  the 
absence  of  strong  quantitative  evidence  to  the  contrary  [10,  11].  The  null  hypothesis  (//„)  for  the 
present  study  may  be  stated  as  follows:  “The  true  mean  difference  between  the  transmissometer 
and  the  DOCS  opacity  measurement  methods  is  greater  than  7.5%”.  This  statement  reflects  the 
assumption  that,  in  the  absence  of  strong  quantitative  data  to  the  contrary,  the  two  opacity 
measurement  methods  are  not  equivalent.  Similarly,  the  alternative  hypothesis  ( Ha )  may  be 
constructed  as  follows:  “The  true  mean  difference  between  the  transmissometer  and  the  DOCS 
opacity  measurement  method  is  equal  to  or  less  than  7.5%”.  The  rationale  for  constructing  the 
null  and  alternative  hypothesis  in  this  fashion  is  to  shift  the  burden  of  proof  for  demonstrating 
Method  9  equivalency  to  the  strength  of  the  DOCS  field  demonstration  data  (Equation  1). 


Null  Hypothesis  ( H0 ):  A;  >7.5% 

Alternative  Hypothesis  (Ha):  So  <7.5%  (1) 

Where  So  =  true  mean  difference  (opacity  measured  using  DOCS  -  opacity  measured 

using  a  transmissometer) 
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In  other  words,  in  the  absence  of  field  data  that  strongly  support  the  rejection  of  the  null 
hypothesis,  the  conclusion  drawn  from  the  data  will  be  that  DOCS  technology  is  not  equivalent 
to  Method  9.  Alternatively,  if  the  strength  of  the  data  is  sufficient  to  reject  the  null  hypothesis 
(acceptance  of  Ha ),  the  conclusion  drawn  from  the  data  will  be  that  the  DOCS  technology  is 
statistically  equivalent  to  Method  9. 

Given  an  assigned  level  of  significance,  a,  and  degrees  of  freedom  (n- 1),  Equations  2  and  3 
define  the  critical  /-value  and  test-statistic  (best),  respectively.  These  parameters  are  compared  to 
detennine  whether  the  strength  of  the  data  is  sufficient  to  reject  the  null  hypothesis  as  depicted 
by  the  statistical  test  condition. 


Critical  t  -  value  :t 


(2) 


Where 


Test  statistic  :ttest 


d  -  S0 


sd 


(3) 


n:  Number  of  paired  measurements  (/.<?.,  DOCS-transmissometer) 

d :  Mean  difference  between  DOCS  and  transmissometer  measurements 

sd 

i—  :  Standard  error 

dn 

Sd :  Square  root  of  variance 


Equation  4  illustrates  the  statistical  test  conditions  used  to  support  or  reject  the  null  hypothesis. 
In  practical  terms,  rejection  of  the  null  hypothesis  is  tantamount  to  accepting  the  DOCS 
technology  as  an  equivalent  method  to  Method  9,  whereas  failure  to  reject  the  null  hypothesis 
essentially  means  that  the  DOCS  technology  is  not  a  statistically  equivalent  method.  Table  4-8 
summarizes  the  results  of  hypothesis  testing  using  data  obtained  from  the  three  EPA-approved 
smoke  schools. 


STATISTICAL  TEST  CONDITIONS 

Test  Condition:  If  t  <ta  ,  then  the  null  hypothesis,  Ho,  is  rejected 

2,nA 

Test  Condition:  If  >ta  ,  data  do  not  support  rejection  of  the  null  hypothesis,  Ho, 

-,n-i 

(accept  Ha ).  (4) 


26 


Table  4-8  Summary  of  Hypothesis  Testing  Performed  at  the  0.05  Significance  Level 


Smoke 

School 

Color  of 
Smoke 

Opacity 

Range  (%) 

Mean 

Difference 

(%) 

Number  of 
Samples 

Significance 

Level 

(a) 

Critical1 1- 
value 

Test  Statistic 

Reject2  Null 
Hypothesis? 

Utah 

Black 

0-100 

6.4 

2236 

0.05 

1.96 

-6.77 

Yes 

Black 

0-60 

5.6 

1957 

0.05 

1.96 

-12.08 

Yes 

Black 

0-40 

5.4 

1745 

0.05 

1.96 

-12.50 

Yes 

White 

0-100 

10.0 

2405 

0.05 

1.96 

10.39 

No 

White 

0-60 

6.7 

1897 

0.05 

1.96 

-4.35 

Yes 

White 

0-40 

5.9 

1686 

0.05 

1.96 

-8.99 

Yes 

Georgia 

Black 

0-100 

8.6 

4949 

0.05 

1.96 

8.06 

No 

Black 

0-60 

8.2 

3620 

0.05 

1.96 

4.49 

No 

Black 

0-40 

7.9 

2896 

0.05 

1.96 

2.22 

No 

White 

0-100 

13.2 

3535 

0.05 

1.96 

19.93 

No 

White 

0-60 

8.5 

2565 

0.05 

1.96 

4.72 

No 

White 

0-40 

7.2 

2203 

0.05 

1.96 

-1.47 

Yes 

Ohio 

Black 

0-100 

10.9 

3498 

0.05 

1.96 

16.83 

No 

Black 

0-60 

9.4 

3066 

0.05 

1.96 

10.24 

No 

Black 

0-40 

8.1 

2753 

0.05 

1.96 

3.53 

No 

White 

0-100 

21.6 

4394 

0.05 

1.96 

46.81 

No 

White 

0-60 

15.0 

3758 

0.05 

1.96 

36.32 

No 

White 

0-40 

12.3 

3131 

0.05 

1.96 

28.67 

No 

'From  standard  statistical  tables,  for  a  =  0.05  and  n>  120,  /-critical  is  approximately 

1.96 

"Where  the  null  hypothesis  is  rejected,  the  data  indicate  that  DOCS  is  equivalent  to 

EPA  Reference  Method  9.  Conversely,  if  the  null  hypothesis  is  not  rejected,  the 
conclusion  is  that  DOCS  is  not  equivalent  to  EPA  Reference  Method  9. 

Significance  testing  yielded  mixed  results  with  respect  to  the  equivalency  of  the  DOCS 
technology  to  Method  9.  In  all  but  one  opacity  range  (high  white  plumes),  the  data  from  the 
Utah  smoke  school  supported  rejection  of  the  null  hypothesis  indicating  that  the  conclusion 
drawn  from  the  data  should  be  that  the  DOCS  technology  is  equivalent  to  Method  9. 

Conversely,  in  the  Georgia  and  Ohio  smoke  schools,  the  DOCS  technology  field  data,  in  general, 
failed  to  support  the  rejection  of  the  null  hypothesis.  Only  in  the  limited  opacity  range  of  0  to 
40%  for  white  smoke  plumes  in  the  Augusta,  Georgia,  field  tests  did  the  data  suggest  that  the 
DOCS  technology  was  statistically  equivalent  to  Method  9.  The  results  for  the  three  smoke 
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school  field  tests  seem  to  indicate  that  climatic  conditions  have  a  profound  effect  on  the  ability  of 
the  DOCS  technology  to  accurately  measure  plume  opacity.  During  the  Utah  field  test,  sky 
conditions  were  clear,  which  provided  optimal  visual  contrast  between  the  smoke  plume  and  sky. 
Under  clear  blue  skies,  the  accuracy  of  the  DOCS  in  measuring  plume 

opacity  was  significantly  better  than  the  established  Method  9  accuracy  standard.  Alternatively, 
when  smoke  plumes  were  viewed  under  weather  conditions  characterized  by  cloudy  or  overcast 
skies,  the  DOCS  technology  has  difficulty  in  accurately  quantifying  opacity.  However,  under 
these  same  conditions,  the  accuracy  of  Method  9-certified  human  observers  in  estimating  visible 
opacity  was  also  severely  diminished  [6]. 

At  the  conclusion  of  Phase  I  of  the  DOCS  technology  field  demonstration,  both  the  EPA’s 
Emission  Measurement  Center  and  the  Optical  Science  Advisory  Panel  strongly  encouraged  the 
development  and  submission  of  a  technical  manuscript  summarizing  all  the  smoke  school  data  to 
a  peer-reviewed  scientific  journal.  Hill  AFB  environmental  management  directorate  accepted 
the  task  of  preparing  and  submitting  the  technical  manuscript,  which  was  published  by  the 
Journal  of  the  Air  &  Waste  Management  Association  (JAWMA)  in  2004  [12]. 

4.3.5  Phase  II — DOCS  Technology  Field  Demonstration 

Mild  temperatures,  overcast  skies  and  precipitation  characterized  the  climate  during  the  DOCS 
technology  Phase  II  field  demonstration  activities.  On-site  field  personnel  recorded  the  values  of 
specific  climatic  parameters  including  1)  mean  air  temperature,  2)  average  wind  speed,  3) 
maximum  wind  speed,  4)  sky  conditions,  5)  relative  humidity,  6)  visibility,  7)  barometric 
pressure,  8)  precipitation,  9)  horizontal  sun  angle  and  10)  vertical  sun  angle.  Table  4-9  provides  a 
summary  of 


Table  4-9  Climatic  Conditions  for  DOCS  Phase  II  Field  Demonstration  in  Alaska 


Parameter 

Day  of  Test 

Day  1 

Day  2 

Mean  Temperature  (°  F) 

54.0 

41.9 

Average  Wind  Speed  (mph) 

4.5 

1.2 

Maximum  Wind  Speed  (mph) 

10.4 

4.6 

Wind  Direction 

Not  measured 

Not  Measured 

Sky  Conditions 

Rain,  Overcast 

Mist,  Overcast 

Relative  Humidity  (percent) 

100 

96 

Visibility  (miles) 

6.0 

3.0 

Barometric  Pressure  (in.  of  Hg) 

29.56 

29.74 

Precipitation  (in.) 

0.82 

0.00 

Vertical  Sun  Angle  (degrees) 

No  sun  visible 

No  sun  visible 

weather  conditions  that  occurred  during  the  two  days  of  the  DOCS  field  testing  in  the  cities  of 
Anchorage  and  Healy,  Alaska,  is  provided  in.  It  should  be  noted  that  the  dark  overcast  skies 
made  it  virtually  impossible  to  visually  detect  and  measure  a  vertical  sun  angle  at  both  the 
Anchorage  and  Healy  DOCS  technology  field  demonstration  sites. 
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4.3.5. 1  Quantitative  Analysis — DoD  Industrial  and  Commercial  Sites 

In  Phase  II,  the  DOCS  technology  was  employed  to  quantity  the  visual  opacity  associated  with 
DoD  industrial  and  commercial  processes  in  the  state  of  Alaska.  Except  for  the  EPA-certided 
smoke  generator  in  Anchorage  and  the  coal-fired  power  plant  in  Healy,  none  of  the  regulated  air 
sources  were  equipped  with  opacity  monitors.  In  the  absence  of  a  COM,  the  opacity  data 
collected  from  these  air  sources  were  used  to  detennine  whether  the  DOCS  technology  and 
Method-9-certified  opacity  readers  yielded  opacity  measurement  results  that  were  statistically 
different.  Table  4-10  summarizes  the  DOCS  technology  field  demonstration  data  from  those  air 
sources  equipped  with  a  COMs. 


Table  4-10  DOCS  Opacity  Measurements  of  Sources  Equipped  with  a  Continuous  Opacity 
Monitor  (COM) 


Opacity  Measurement 
Approach 

Opacity 

Range  (%) 

Mean 

Difference  (%) 

Number  of 
Samples 

99%  Cl1 

DOCS— COM 

0-100% 

14.1 

215 

11.6-16.6 

Certified  observers — COM 

0-100% 

6.0 

224 

5.2-6. 8 

'99%  Cl  =  99%  confidence  interval 


Table  4-11  summarizes  the  statistical  comparison  between  the  mean  DOCS  opacity  measurement 
and  the  mean  opacity  measurement  recorded  by  Method-9-certified  human  observers  at  regulated 
air  sources  where  there  was  no  COM  installed.  When  the  comparison  was  made  using  a  limited 
opacity  range  of  0  to  40%  (based  on  Method-9-certified  reader  measurements),  the  mean  opacity 
difference  between  the  DOCS  technology  and  certified  human  observers  decreased  to  5.0  %. 
These  data  suggest  that  the  difference  between  the  two  methods  in  measuring  visible  opacity  is 
significant  and  that,  on  average,  the  DOCS  tends  to  read  visible  emissions  at  higher  opacity 
levels  than  Method-9-certified  human  observers  [13]. 


Table  4-11  Statistical  Data  Summary  of  Alaska  Field  Measurements  without  COM 


Opacity  Measurement 
Approach 

Opacity 
Range  (%) 

Mean  Difference 
(%) 

Number  of 
Samples 

99%  Cl1 

DOCS — Certified  observers 

0-100 

10.5 

360 

8.9-12.2 

DOCS — Certified  observers 

0-40 

5.0 

255 

3. 7-6. 3 

'99%  Cl  =  99%  confidence  interval 


With  respect  to  achieving  the  technology  accuracy  performance  criteria,  Phase  I  and  II  field 
demonstration  results  confirmed  that  the  DOCS  technology  consistently  achieved  the  federally 
codified  Method  9  accuracy  standard  when  the  technology  was  employed  under  clear  ( i.e .,  blue 
sky)  weather  conditions.  Moreover,  using  the  width  of  the  99%  confidence  about  the  mean 
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opacity  difference  as  a  metric  for  reliability,  the  field  demonstration  results  confirmed  that  the 
reliability  of  the  DOCS  technology  exceeded  that  of  Method  9  for  all  types  of  smoke  plumes  and 
under  all  weather  conditions.  The  superior  reliability  of  the  DOCS  technology  was  not 
surprising  given  the  fact  that  estimation  of  visible  opacity  using  a  digital  camera  based  system  is 
much  less  subjective  than  employing  human  observers. 

Although  the  DOCS  technology  was  inconsistent  in  achieving  the  codified  Method  9  accuracy 
standard  when  evaluated  under  adverse  weather  conditions,  the  regulatory  approved  opacity 
measurement  approach  (i.e.,  Method  9-certified  human  observers)  consistently  failed  to  maintain 
the  accuracy  standard  when  evaluated  under  these  same  climatic  conditions.  These  findings 
suggest  important  technical  limitations  and  considerations  for  DoD  personnel  tasked  with 
measuring  and  reporting  visual  opacity.  First,  the  DOCS  technology  would  be  the  preferred 
approach  over  Method-9-certified  readers  when  measuring  visible  opacity  under  fair  weather 
(< e.g . ,  clear  sky)  conditions.  Secondly,  under  adverse  weather  conditions,  the  diminished 
measurement  accuracy  experienced  by  the  DOCS  technology  is  comparable  to  the  reduced 
measurement  accuracy  documented  by  Method-9-certified  human  observers.  Despite  its 
reduced  accuracy  under  poor  weather  conditions,  its  superior  reliability  suggests  that  the  DOCS 
technology  would  be  the  superior  opacity  measurement  method  under  all  weather  conditions. 
Third,  the  poor  performance  of  the  Method-9-certified  human  observers  in  estimating  visible 
opacity  under  overcast  sky  conditions  clearly  illustrates  not  only  that  the  current  federally 
approved  opacity  measurement  approach  is  subjective  but  that  success  in  completing  Method  9 
certification  training  provides  little  assurance  that  the  Method  9-certified  human  observer  can 
estimate  the  opacity  of  regulated  air  sources  with  any  known  degree  of  confidence. 

Finally,  owing  to  both  the  scientific  and  regulatory  significance  of  the  Phase  II  DOCS  field 
demonstration  activities,  the  EPA’s  Emission  Measurement  Center  in  conjunction  with  the 
Opacity  Science  Advisory  Panel  strongly  recommended  that  the  key  findings  of  the  DOCS  Phase 
II  demonstration  activities  be  summarized  and  published  in  a  peer-reviewed  journal  article.  Hill 
AFB  environmental  management  directorate  accepted  the  responsibility  of  preparing  and 
submitting  a  technical  manuscript  of  the  DOCS  Phase  II  field  demonstration,  which  was 
published  by  the  Federal  Facilities  Environmental  Journal  in  2004  [13]. 
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5. 


Cost  Assessment 


5.1  Cost  Reporting 

Tables  5-1  and  5-2  summarize  the  estimated  costs  associated  with  the  implementation  of  Method 
9  at  a  DoD  stateside  and  remote  location,  respectively.  Although,  in  principle,  there  are  no 
specifically  required  Method  9  equipment  costs,  in  practice,  the  Method  9  visual  opacity  reader 
must  purchase  or  at  least  have  access  to  a  range  of  climatic  monitoring  equipment  including  the 
following  1)  anemometer,  2)  sling  psychrometer,  3)  Abney  Level  (sun  angle  measurement),  4) 
stop  watch  and  5)  magnetic  compass.  The  total  purchase  price  for  these  instruments  is 
approximately  $2,000.00,  which  does  not  include  an  annual  equipment  maintenance  costs 
estimated  at  $200.00. 


Table  5-1  Method  9  Implementation  Costs  at  a  Stateside  DoD  Facility 


Direct  Environmental 

Activity  Process  Cost 

Indirect 
Environmental 
Activity  Costs 

Other  Costs 

Start-Up 

Operation  and 
Maintenance 

Activity 

$  K 

Activity 

$  K 

Activity 

$  K 

Activity 

Method  9 
Equipment 
purchase 

0 

Consumables 
and  Supplies 

0.025 

Compliance 

Audits 

0 

Overhead 
Associated 
With  Process 

0 

Equipment 

Design 

NA 

Equipment 

Maintenance 

0 

Document 

Maintenance 

0.06 

Productivity/ 
Cycle  Time 

2.4 

Training 

operators 

4.1 

Training  of 
Operators 

8.2 

Environmental 

Management 

Plan 

Maintenance 

0.06 

Worker  Injury 
Claims  & 
Health  Costs 

0 

Environmental 

Monitoring 

Equipment 

2.0 

Environmental 

Monitoring 

Equipment 

0.2 

Reporting 

Requirements 

0.12 

SUMMARY  OF  COSTS 

Total  Initial 
Investment 
Costs' 

1 

Total  Direct 
Operation 
and 

Maintenance 

Costs 

8.425 

Total  Indirect 
Environmental 
Activity  Costs 

0.24 

Total  -  Other 
Costs 

2.4 

Total  Annual 
Operating 
Costs2  (8.425 
+  0.24  +  0.24) 

8.905 

'Total  Initial  Investment  Costs  consist  of  only  Start-Up  Costs 

2Total  Annual  Operating  Costs  consist  of  Total  Direct  O&M,  Total  Indirect  Environmental 
Activity  Costs  and  Other  Costs 
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Method  9  certification  training  costs  were  estimated  to  be  approximately  $4,100.00  per  stateside 
facility  trainee  and  $6,300.00  per  trainee  located  at  a  remote  DoD  facility.  The  relatively  large 
difference  in  training  costs  between  stateside  and  remote  DoD  locations  is  directly  attributable  to 
the  need  to  hire  temporary  contract  labor  at  remote  facilities  to  cover  assigned  worker  duties 
when  full  time  employees  are  attending  Method  9  training. 

The  Method  9  training  requires  that  the  trainees  undergo  a  three-(3)-day  training  program  that 
basically  consists  of  one  day  of  classroom  instruction  followed  by  two  days  of  opacity  field 
testing.  For  the  stateside  facilities,  the  Method  9  training  costs  for  a  minimum  of  two  (2) 
individuals  consists  of  the  following  items:  1)  $350.00  tuition  per  attendee,  2)  labor  costs  at 
$  15.00/hr  for  40  hours  and  3)  $1 100.00  travel  costs  per  attendee.  For  DoD  personnel  stationed 
at  remote  locations,  the  Method  9  training  cost  for  a  minimum  of  two  (2)  individuals  consists  of: 
1)  $350.00  tuition  per  attendee,  2)  labor  costs  at  $  15.00/hr  for  40  hours,  3)  replacement  labor 
costs  at  $  15.00/hr  for  40  hours,  4)  $1 100.00  travel  costs  per  attendee  and  5)  $1 100.00  travel  costs 


Table  5-2  Method  9  Implementation  Costs  at  a  Remote  DoD  Facility 


Direct  Environmental  Activity  Process  Cost 

Indirect 
Environmental 
Activity  Costs 

Other  Costs 

Start-Up 

Operation  and 
Maintenance 

Activity 

$  K 

Activity 

$  K 

Activity 

$  K 

Activity 

$  K 

Method  9 
Equipment 
purchase 

0 

Consumables 
and  Supplies 

0.025 

Compliance 

Audits 

0 

Overhead 
Associated 
with  Process 

0 

Equipment 

Design 

NA 

Equipment 

Maintenance 

0 

Document 

Maintenance 

0.06 

Productivity/ 
Cycle  Time 

4.8 

Training 

Operators 

6.3 

Training  of 
Operators 

12.6 

Environmental 

Management 

Plan 

Maintenance 

0.06 

Worker  Injury 
Claims  & 
Health  Costs 

0 

Environmental 

Monitoring 

Equipment 

2.0 

Environmental 

Monitoring 

Equipment 

0.2 

Reporting 

Requirements 

0.12 

j  SUMMARY  OF  COSTS 

Total  Initial 
Investment 
Costs1 

8.3 

Total  Direct 
O&M  Costs 

12.83 

Total  Indirect 
Environmental 
Activity  Costs 

0.24 

Total  -  Other 
Costs 

4.8 

Total  Annual 
Operating 
Costs2  (12.825 
+  0.24  +  0.48) 

13.55 

2Total  Annual  Operating 
Activity  Costs  and  Other 


Costs  consist  of  Total  Direct  O&M,  Total  Indirect  Environmental 
Costs 


32 


per  replacement  labor.  Indirect  costs,  which  include  environmental  document  storage  and 
maintenance,  are  estimated  to  be  approximately  $240.00  per  year. 


Productivity/cycle  time  costs  are  associated  with  worker  productivity  losses  directly  attributable 
to  a  worker  receiving  salary  but  essentially  being  absent  from  official  duties  during  Method  9 
training.  At  stateside  facilities,  the  financial  value  of  worker  productivity  loss  associated  with 
Method  9  is  anticipated  to  be  $2,400.00.  This  estimate  consists  of  the  costs  associated  with 
individuals  being  absent  from  their  full-time  work  assignment  for  40  hours  at  a  pay  rate  of 
$  15.00/hr  twice  per  year.  Similarly,  for  DoD  personnel  stationed  at  remote  sites,  productivity/ 
cycle  time  costs  are  estimated  to  be  approximately  $4,800.00.  This  estimate  includes  the  costs 
associated  with  full-time  employees  (2)  and  temporary  hires  (2)  being  absent  from  their  full-time 
work  assignment  for  40  hours  at  a  pay  rate  of  $15. 00/hr  twice  per  year.  Consumables  and 
supplies  in  support  of  Method  9  measurements  are  estimated  to  be  $25.00  per  year. 


Table  5-3  DOCS  Implementation  Costs  at  a  Stateside  DoD  Facility 


Direct  Environmental  Activity  Process  Cost 

Indirect 
Environmental 
Activity  Costs 

Other  Costs 

Start-Up 

Operation  and 
Maintenance 

Activity 

$  K 

Activity 

$  K 

Activity 

$  K 

Activity 

$  K 

DOCS 

Equipment 

purchase 

3.00 

Consumables 
and  Supplies 

0.075 

Compliance 

Audits 

0 

Overhead 
Associated 
with  Process 

0 

Equipment 

Design 

NA 

Equipment 

Maintenance 

0.10 

Document 

Maintenance 

0.06 

Productivity/ 
Cycle  Time 

0.48 

Training 

Operators 

3.14 

Training  of 
Operators 

0.50 

Environmental 

Management 

Plan 

Maintenance 

0.06 

Worker 
Injury 
Claims  & 
Health  Costs 

0 

Environmental 

Monitoring 

Equipment 

2.00 

Environmental 

Monitoring 

Equipment 

0.20 

Reporting 

Requirements 

0.12 

SUMMARY  OF  COSTS 

Total  Initial 
Investment 
Costs1 

8.14 

Total  Direct 
O&M  Costs 

0.88 

Total  Indirect 
Environmental 
Activity  Costs 

0.24 

Total  - 
Other  Costs 

4.8 

Total  Annual 
Operating 
Costs2  (0.875 
+  0.24  +  0.48) 

5.92 

'Total  Initial  Investment  Costs  consist  of  only  Start-Up  Costs 

2Total  Annual  Operating  Costs  consist  of  Total  Direct  O&M,  Total  Indirect  Environmental 
Activity  Costs  and  Other  Costs 
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Tables  5-3  and  5-4  summarize  the  estimated  implementation  costs  for  supporting  a  DOCS 
opacity  measurement  program  for  a  single  regulated  source  at  a  state-side  and  remote  DoD 
location,  respectively.  The  capital  costs  for  the  DOCS  technology  is  approximately  $3,000.00 
and  includes  the  purchase  of  a  medium-end  digital  camera  with  five-year  service  contract 
(@  $750.00),  laptop  computer  with  five-year  service  contract  (@  $1,500.00),  DOCS  software 
license  (@  500.00)  and  instructional  CD  together  with  other  accessories  (@  $250.00). 

Although,  in  theory,  these  costs  represent  one-time  sunk  costs,  in  reality,  digital  camera  and 
computer  software  technology  continue  to  evolve  and  improve.  For  the  DOCS  technology  to 
take  full  advantage  of  future  scientific  advancements,  it  is  anticipated  that  DOCS  software 
version  releases  compatible  with  improved  digital  camera  hardware  will  be  developed  and  made 
available  to  purchasers  of  DOCS  software  licenses. 


Table  5-4  DOCS  Implementation  Costs  at  a  Remote  DoD  Facility 


Direct  Environmental  Activity  Process 
Cost 

Indirect 
Environmental 
Activity  Costs 
(Annual) 

Other  Costs 
(Annual) 

Start-Up 

Operation  and 
Maintenance 

Activity 

$  K 

Activity 

$  K 

Activity 

$  K 

Activity 

$  K 

Equipment 

purchase 

3.00 

Consumables 
and  Supplies 

0.075 

Compliance 

Audits 

0 

Overhead 
Associated 
with  Process 

0 

Equipment 

Design 

NA 

Equipment 

Maintenance 

0.10 

Document 

Maintenance 

0.06 

Productivity 
/Cycle  Time 

0.48 

Training 

Operators 

6.06 

Training  of 
Operators 

0.50 

Environmental 

Management 

Plan 

Preparation  and 
Maintenance 

0.06 

Worker 
Injury 
Claims  & 
Health 
Costs 

0 

Environmental 

Monitoring 

Equipment 

2.00 

Environmental 

Monitoring 

Equipment 

0.20 

Reporting 

Requirements 

0.12 

SUMMARY  OF  COSTS 

Total  Initial 
Investment 
Costs1 

11.06 

Total  Direct 
O&M  Costs 

0.88 

Total  Indirect 
Environmental 
Activity  Costs 

0.24 

Total  - 

Other 

Costs 

4.8 

Total  Annual 
Operating 
Costs2  (0.875 
+  0.24  +  0.48) 

5.92 

'Total  Initial  Investment  Costs  consist  of  only  Start-Up  Costs 

2Total  Annual  Operating  Costs  consist  of  Total  Direct  O&M,  Total  Indirect  Environmental 
Activity  Costs  and  Other  Costs 
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In  addition  to  the  DOCS  capital  costs,  the  DOCS  user  must  purchase  or  at  least  have  access  to  a 
range  of  climatic  monitoring  equipment  including  the  following:  1)  anemometer,  2)  sling 
psychrometer,  3)  Abney  Level  (sun  angle  measurement),  4)  stop  watch  and  5)  magnetic  compass. 
These  instruments  are  required  for  monitoring  and  documenting  weather  conditions,  which  are 
legally  reportable  information  associated  with  current  Method  9  opacity  compliance  verification 
activities.  It  is  assumed  that  air  quality  regulators  will  continue  to  require  opacity  reporting  to 
be  accompanied  by  relevant  climatic  data  regardless  of  the  opacity  measurement  approach.  The 
purchase  price  for  these  instruments  is  approximately  $2,000.00  and  annual  equipment 
maintenance  costs  are  estimated  to  be  approximately  $200.00.  Consumables  and  supplies  in 
support  of  the  DOCS  technology  hardware  are  estimated  to  be  approximately  $75.00  per  year 
and  include  such  items  as  batteries  and  digital  camera  memory  chips. 

The  costs  associated  with  implementation  of  DOCS  at  state-side  and  remote  DoD  locations  are 
essentially  equivalent  except  for  a  substantial  difference  attributable  to  the  hiring  of  temporary 
workers  for  the  smaller  and  remotely  located  facilities.  When  travel  and  per  diem  costs  are 
taken  into  account,  the  estimated  initial  DOCS  training  costs  for  a  stateside  DoD  facility  are 
estimated  to  be  approximately  $3,140.00.  The  costs  elements  comprising  this  estimate  consists 
of  the  following:  1)  trainee  tuition  at  $350.00,  2)  labor  cost  at  $  15/hr  for  eight  (8)  hours,  3)  travel 
costs  at  $1 100.00  (hotel  at  $300.00,  per  diem  at  $105.00,  $700.00  average  airline  fare)  and  4) 
two  individuals  being  trained.  It  is  assumed  that  at  stateside  DoD  facilities,  a  sufficient  labor 
force  exists  to  cover  the  duties  of  those  individuals  who  are  absent  to  attend  DOCS  training. 

For  a  remotely  located  facility,  the  addition  of  hiring  replacement  workers  increases  the  overall 
training  cost  to  approximately  $6,060.  This  estimate  reflects  the  following  cost  elements:  1) 
trainee  tuition  at  $350.00,  2)  labor  cost  at  $  15/hr  for  eight  (8)  hours,  3)  travel  cost  at  $1 100.00  for 
both  attendee  and  replacement  workers,  4)  replacement  labor  costs  (3  days  or  24  hours  at 
$  15.00/hr)  and  5)  two  individuals  receiving  DOCS  training. 

It  is  anticipated  that  DOCS  training  will  take  no  more  than  eight  (8)  hours,  of  which  the  first  half 
of  the  training  session  will  focus  on  classroom  presentations  of  the  scientific  theory  upon  which 
DOCS  is  based  as  well  as  a  review  of  the  fundamental  operation  of  digital  cameras  and 
photographic  principles.  The  second  half  of  training  will  be  dedicated  to  utilizing  the  DOCS 
software  to  estimate  the  opacity  of  a  series  of  digital  photographs  of  air  sources  of  known  opacity 
possibly  followed  by  a  short  computer  based  examination. 

Once  the  initial  DOCS  technology  training  is  completed,  re-certification  training  can  be 
conducted  for  both  DoD  state-side  and  remotely  located  personnel  through  either  a  web-based  or 
compact  disc  (CD)  training  program.  The  anticipated  cost  for  this  training  is  estimated  to  be 
approximately  $500.00  per  facility.  It  is  also  assumed  that  the  DOCS  operation  and 
maintenance  training  costs  includes  uninterrupted  access  to  a  DOCS  help  desk.  The  DOCS  help 
desk,  which  presumably  will  be  operated  by  a  private  contractor,  will  provide  the  user  with 
technical  assistance  as  well  as  appropriate  version  updates  to  the  DOCS  computer  software. 

The  annual  equipment  operations  and  maintenance  costs  are  estimated  to  be  approximately 
$100.00,  which  includes  the  costs  for  any  minor  repairs  associated  with  either  the  digital  camera 
and/or  the  laptop  computer  (note  that  it  is  assumed  that  any  major  hardware  repairs  would  be 
covered  by  service  contracts  included  in  the  item’s  original  purchase  price).  Indirect  costs, 
which  include  environmental  document  storage  and  maintenance,  is  estimated  to  be 
approximately  $240.00  per  year.  At  both  large  and  remote  facilities,  worker  productivity  loss 
associated  with  DOCS  web-based  or  CD  training  is  estimated  at  $480.00,  which  consists  of  two 
individuals  being  trained  for  eight  (8)  hours  at  a  pay  rate  of  $  1 5 .00/hr  twice  per  year. 
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5.2  Cost  Analysis 

To  provide  a  consistent  basis  upon  which  to  compare  life-cycle  technology  implementation  costs, 
the  data  provided  in  Tables  5-1  through  5-4  are  utilized  to  estimate  the  total  annual  costs 
associated  with  using  either  DOCS  or  Method  9  (Table  5-5).  The  timeframe  over  which  the  costs 
will  be  compared  is  five  (5)  years,  which  is  considered  a  reasonable  useful  life  for  digital 
cameras  and  laptop  computers.  The  assumed  interest  (or  discount)  rate  used  in  developing  the 
annual  financial  estimates  is  4.0%  [14]. 

To  convert  the  one-time  start-up  activity  costs  to  an  annual  (i.e.,  amortized)  cost,  an  equal- 
payment  capital  recovery  factor  (CRF)  is  utilized.  The  value  of  the  CFR  at  4.0%  for  a  five-year 
period  is  approximately  0.2246  [14].  To  estimate  the  total  annual  costs  for  implementation  of 
either  method,  other  annual  cost  elements  are  added  to  the  amortized  start-up  costs.  For 
example,  the  start-up  costs  for  implementation  of  the  DOCS  at  stateside  DoD  facilities  (Table  5- 
3)  is  estimated  to  be  $8,140.00  ($3,000.00  for  DOCS  hardware,  $3,140.00  for  initial  operator 
training  and  $2,000.00  for  environmental  measurement  equipment).  Amortization  of  these  costs 
yields  an  annual  cost  of  $1,828.24  ($8,140.00  •  0.2246).  When  the  other  annual  cost  elements 
are  added  to  the  amortized  start-up  cost,  the  total  annual  cost  for  implementing  and  maintaining  a 
DOCS  system  at  a  state-side  facility  is  estimated  to  be  $3,423.24. 

Results  from  Table  5-5  illustrate  that  DOCS  implementation  results  in  substantial  cost  savings 
compared  to  Method  9  at  both  stateside  and  remote  locations.  The  magnitude  of  the  potential 
financial  benefits  associated  with  employing  DOCS  will  vary  with  DoD  facility  size  and  the 
number  of  opacity  inspectors  currently  employed  at  the  facility.  For  example,  the  annual  cost 
savings  associated  with  implementation  of  DOCS  on  stateside  facilities  is  approximately 
$9,01 1.82  ($12,435.06-  $3,423.24)  per  pair  of  trained  technology  users.  Given  that  DoD 
currently  supports  Method  9  certification  for  approximately  3,400  opacity  readers  (i.e.,  1,700 
pairs  of  opacity  readers),  the  aggregate  annual  savings  to  DoD  in  adopting  DOCS  would  be 
approximately  15.3  million  dollars  per  year.  Accounting  of  the  annual  cost  savings  associated 
with  the  implementation  of  DOCS  on  remote  DoD  facilities  would  yield  even  larger  annual  cost 
savings. 


Table  5-5  Total  Annual  Cost  Estimates  for  Implementing  DOCS  and  Method  9 


IMPLEMENTATION 

SCENARIO 

Amortized 

Start-Up 

Costs' 

Annual 

O&M  Costs 

Indirect 
Environmental 
Activity  Costs 

Productivity 
/Cycle  Time 
Costs 

TOTAL 

ANNUAL 

COSTS2 

DOCS;  stateside 

$1,828.24 

$875.00 

$240.00 

$480.00 

$3,423.24 

DOCS;  remote  location 

$2,484.08 

$875.00 

$240.00 

$480.00 

$4,079.08 

Method  9;  stateside 

$1,370.06 

$8,425.00 

$240.00 

$2,400.00 

$12,435.06 

Method  9;  remote 
location 

$1,864.18 

$12,825.00 

$240.00 

$4,800.00 

$19,729.18 

'Amortized  costs  are  based  on  4%  discount  rate  applied  over  a  five-year  period — Capital 

Recovery  Factor  is  approximately  0.2246 

2Total  annual  costs  are  per  pair  of  trained  individuals 
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A  net  present  value  (NPV)  analysis  was  employed  to  estimating  total  life-cycle  costs  savings. 
Using  a  discount  rate  of  4%,  the  estimated  present  worth  factor  (PWF)  for  a  five-year  product  life 
is  approximately  4.45 18  [14].  Multiplying  the  PWF  by  the  annual  cost  savings  yields  the  NPV 
for  each  implementation  scenario.  The  difference  between  the  NPVs  for  the  alternatives  is 
essentially  the  life-cycle  cost  savings.  For  example,  the  NPV  for  DOCS  and  Method  9  stateside 
implementation  are  $15,239.58  ($3,423.24  •  4.4518)  and  $55,358.40  ($12,435.06  •  4.4518), 
respectively,  per  pair  of  trained  technology  users.  The  difference  between  these  two  values, 

$40,1 18.82,  reflects  the  life-cycle  cost  savings  per  pair  of  trained  DOCS  technology  users.  It 
should  be  noted  that  if  all  DoD  supported  visual  opacity  readers  were  assumed  to  be  stationed  at 
stateside  facilities,  adoption  of  the  DOCS  technology  has  the  potential  of  saving  DoD  in  excess 
of  68.2  million  dollars  ($40,1 18.82  per  pair  of  trained  DOCS  technology  users  •  1,700  pairs  of 
opacity  readers  in  all  of  DoD)  over  the  useful  life  of  the  DOCS  equipment  (estimated  at  five 
years).  Table  5-6  summarizes  both  annual  cost  savings  and  the  life-cycle  cost  savings  associated 
with  DoD  adoption  of  the  DOCS  technology  per  pair  of  trained  DOCS  technology  users. 


Table  5-6 


Annual  Cost  and  Life-Cycle  Cost  Savings  Associated  with  Implementing  DOCS 


i 


Annual  Cost  Savings 
to  Implement  DOCS 
at  Stateside  DoD 
Location2 

Annual  Cost  Savings 
to  Implement  DOCS 
at  Remote  DoD 
Location3 

Life-Cycle  Cost 
Savings  to  Implement 
DOCS  at  Stateside 
DoD  Location 

Life-Cycle  Cost  Savings 
to  Implement  DOCS  at 
Remote  DoD  Location 

$9,011.82 

$15,650.10 

$40,118.82 

$69,671.12 

’Cost  basis  is  per  pair  of  trained  technology  users  at  a  facility 
2($  12,435.06  -  $3,423.24)  see  Table  5-5 
3($19, 729.18  -  $4,079.08)  see  Table  5-5 


Finally,  the  ability  of  the  DOCS  technology  to  archive  digital  photographs  of  visible  opacity 
represents  another  potential  cost  savings  to  regulated  DoD  facilities.  In  principle,  this  ability 
improves  the  forensic  reliability  of  the  opacity  measurement,  which  can  potentially  reduce  the 
number  of  legal  enforcement  actions  that  a  regulatory  agency  may  take  against  a  facility.  With  a 
significant  number  of  DoD  air  sources  required  to  operate  within  a  specified  opacity  range, 
providing  regulatory  agencies  with  an  opacity  measurement  accompanied  by  a  digital  photograph 
of  the  source  during  operation  would  be  more  compelling  evidence  of  the  actual  opacity  level 
and  provide  irrefutable  evidence  of  the  source’s  compliance  status.  Moreover,  regulatory 
agencies  could  analyze  the  digital  photograph  at  any  time  using  the  DOCS  software  (or 
equivalent  approach)  to  verify  the  reported  opacity  level. 


5.3  Cost  Comparison 

As  illustrated  in  Tables  5-5  and  5-6,  the  DOCS  technology  represents  a  financially  competitive 
alternative  to  either  Method  9  or  a  continuous  opacity  monitor  (COM)  for  measuring  the  visual 
opacity  of  regulated  air  sources.  Method  9  compliance  costs  consist  of  budgeting  resources  for 
training  individuals  every  six  months.  Given  the  Method  9  training  and  lost  worker  productivity 
costs  cited  in  Table  5-5,  and  the  fact  that  DoD  currently  certifies  3,400  individuals  in  Method  9, 
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the  resource  level  required  to  support  Method  9  across  the  entire  DoD  is  estimated  at 
approximately  $21.1  million  dollars  per  year  (stateside  costs  at  $12,435.06  per  pair  of  trained 
individuals  •  1,700  pairs  of  trained  individuals).  If  the  percentage  of  Method  9-certified 
individuals  who  are  actually  stationed  at  remote  locations  were  taken  into  account,  the  full  cost 
of  supporting  a  Method  9  certification  program  would  be  much  greater  than  the  $21.1  million 
dollars  per  year  estimate. 

In  contrast,  the  annual  costs  to  implement  and  maintain  the  DOCS  technology  are  approximately 
76%  less  than  those  required  to  implement  and  maintain  Method  9  certification.  Therefore, 
adoption  of  DOCS  across  the  DoD  has  the  potential  of  saving  approximately  15.3  million  dollars 
per  year  in  compliance  costs.  As  a  third  option,  a  facility  may  opt  to  install  and  operate  a  COM 
at  a  substantially  higher  cost  than  either  Method  9  or  DOCS.  Purchase  prices  for  a  single  COM 
unit  can  vary  from  $25,000.00  to  $100,000.00  per  source.  Moreover,  at  DoD  facilities 
employing  a  COM,  an  alternative  opacity  measurement  approach  is  typically  required  during 
those  times  when  the  COM  experiences  a  breakdown  or  is  otherwise  inoperable. 

Acceptance  of  the  DOCS  technology  as  an  approved  regulatory  alternative  to  Method  9  is  the 
largest  and  most  important  cost  driver  in  all  of  these  comparisons.  The  future  of  the  DOCS 
technology  and  all  other  digital-camera-based  opacity  measurement  technologies  is  highly 
sensitive  to  the  development  and  promulgation  of  an  EPA-approved  digital-camera-based  opacity 
verification  test  method.  Beyond  the  financial  impact  stemming  from  regulatory  approval  of  the 
DOCS,  better-quality  and  lower-cost  digital  equipment  is  continuously  being  introduced  into  the 
marketplace.  If  the  DOCS  technology  improvements  can  be  supported  to  take  advantage  of  new 
scientific  and  product  development,  user  costs  should  continue  to  decrease.  Finally,  it  should  be 
noted  that  the  DOCS  technology  contains  no  inherent  hazardous  chemicals  and  system  disposal 
requires  no  unique  methodology. 
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6.  Implementation  Issues 


6.1  Cost  Observations 

The  overarching  factor  that  affected  project  demonstration  costs  was  regulatory  acceptance  of  the 
technology.  Although  the  DOCS  technology  is  relatively  inexpensive  to  purchase  and 
implement,  many  facilities  that  would  have  been  ideal  demonstration  sites  were  reluctant  to 
allow  the  DOCS  team  to  evaluate  their  regulated  air  emission  sources  with  an  experimental 
device  that  had  yet  to  receive  regulatory  approval.  To  identify  and  gain  facility  permission  to 
apply  the  technology  at  regulated  facilities  required  lengthy  negotiations  as  well  as  regulatory 
assurances  that  participating  facilities  would  not  be  susceptible  to  enforcement  action  based  on 
the  DOCS  field  demonstration  results.  It  is  anticipated  that  once  final  EPA  approval  of  a  digital- 
camera  based  opacity  measurement  method  is  established,  legal  concerns  expressed  by  the 
regulated  community  regarding  participation  in  future  DOCS  field  demonstrations  will  be 
minimized,  if  not  eliminated. 

Beyond  the  cost  impacts  associated  with  the  lack  of  regulatory  approval  of  DOCS,  poor  weather 
conditions  had  an  impact  on  demonstration  costs.  Under  ideal  conditions,  opacity  measurements 
are  typically  conducted  under  weather  conditions  characterized  by  light  to  no  wind  and  little  to 
no  precipitation.  In  the  current  demonstration,  the  field  demonstration  schedule  was  established 
months  before  actual  testing  occurred.  At  many  sites,  opacity  measurements  were  made  under 
adverse  weather  conditions  ( e.g .,  high  wind,  heavy  precipitation,  dark  overcast  skies,  etc.). 

When  possible,  attempts  were  made  to  delay  the  collection  of  field  data  until  weather  conditions 
improved  (which  increased  costs).  In  other  circumstances,  personnel  travel  schedules  did  not 
permit  the  delaying  of  the  data  collection  activities. 


6.2  Performance  Observations 

With  respect  to  the  primary  perfonnance  criteria,  results  from  the  field  demonstration  illustrated 
that  the  DOCS  consistently  achieved  the  federally  codified  Method  9  accuracy  standard  when  the 
technology  was  employed  under  blue  sky  weather  conditions.  Although  the  DOCS  was 
inconsistent  in  achieving  the  primary  perfonnance  criteria  when  evaluated  under  adverse  weather 
conditions,  the  regulatory  approved  opacity  approach  (i.e.,  use  of  Method  9-certified  human 
observers)  consistently  failed  to  achieve  the  accuracy  standard  when  evaluated  under  these  same 
climatic  conditions.  These  findings  clearly  illustrate  that  the  current  federally  approved  opacity 
measurement  approach  (i.e.,  use  of  Method  9-certified  human  observers)  is  not  only  subjective 
but  that  success  in  completing  Method  9  certification  training  provides  little  assurance  that  the 
Method  9-certified  human  observer  can  estimate  the  opacity  of  regulated  air  sources  with  any 
known  degree  of  confidence. 

The  secondary  performance  criteria  focused  on  the  reliability  of  DOCS  relative  to  Method  9- 
certified  human  observers.  Using  the  width  of  the  99%  confidence  about  the  mean  opacity 
difference  as  a  metric  for  reliability,  the  field  demonstration  results  confirmed  that  the  DOCS 
performance  exceeded  that  of  Method  9  under  all  types  of  smoke  plumes  and  weather  conditions. 
The  superior  reliability  of  the  DOCS  technology  was  not  surprising  given  the  fact  that  estimation 
of  visible  opacity  using  a  digital  camera  based  system  is  much  less  subjective  than  employing 
human  observers. 
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6.3  Other  Significant  Observations 

After  a  comprehensive  technical  review  of  the  DOCS  field  demonstration  results  and  in  light  of 
continuous  appeals  by  EPA  regional  offices,  state  regulatory  agencies  and  the  regulated 
community  (including  DoD  facilities)  for  the  establishment  of  a  new  EPA-approved  visible 
opacity  field  measurement  method  as  an  alternative  to  Method  9,  the  EPA’s  Emission 
Measurement  Center  (EMC)  recommended  that  the  DOCS  technical  team  formulate  and  conduct 
a  one-year  DOCS  regulatory  pilot  study  in  which  the  DOCS  would  be  evaluated  side-by-side 
with  Method-9-certified  human  observers  under  regulatory  enforcement  conditions.  The  DOCS 
regulatory  pilot  study,  which  is  due  to  be  completed  in  the  spring  of  2005,  involved  quantifying 
the  visible  opacity  of  regulated  air  sources  located  at  Fort  Hood,  Texas,  Hill  AFB,  Utah,  Fort 
Wainwright,  Alaska,  as  well  as  a  range  of  regulated  air  sources  associated  with  government  and 
private  industrial  entities  currently  operating  within  the  state  of  Utah. 

The  field  results  from  the  DOCS  one-year  regulatory  pilot  study  will  not  only  provide  additional 
DOCS  field  performance  data  collected  under  regulatory  enforcement  conditions  but  will  also 
serve  as  the  technical  basis  upon  which  a  new  EPA-approved  digital-camera-based  visible 
opacity  measurement  method  will  be  developed.  It  is  anticipated  that  a  draft  digital-camera- 
based  opacity  method  will  be  available  for  EPA  review  and  consideration  at  the  conclusion  of  the 
DOCS  regulatory  pilot  study. 

Finally,  the  potential  application  of  DOCS  for  monitoring  and  enforcing  opacity  standards 
associated  with  fugitive  emissions  has  not  escaped  the  notice  of  federal  and  state  regulatory 
agencies  as  well  as  many  DoD  facilities.  Because  of  its  successful  deployment  in  the 
measurement  of  opacity  associated  with  stationary  sources,  the  EPA’s  Emission  Measurement 
Center  (EMC)  has  encouraged  its  DoD  partners  to  consider  developing  and  implementing  a 
DOCS  field  demonstration  program  in  which  the  technology  may  be  applied  to  support  DoD 
environmental  compliance  requirements  associated  with  DoD  operational  and  test  range 
activities  as  well  as  the  military  operation  of  marine  vessels. 


6.4  Lessons  Learned 

For  the  potential  user  of  the  DOCS  technology,  there  were  several  important  lessons  learned  that 
will  impact  technology  implementation.  The  first  is  that  it  must  be  recognized  that  digital 
camera  technology  is  changing  rapidly.  The  two  medium-priced  digital  cameras  that  were 
employed  in  the  DOCS  field  demonstration — Kodak  DC265  and  Kodak  DC290 — are  no  longer 
commercially  available.  Similar-priced  digital  cameras  today  have  much  improved  optical 
qualities  including  enhanced  picture  resolution  as  well  as  long-range  zoom  and  faster  data 
transfer  capabilities.  To  ensure  that  future  DOCS  users  will  have  access  to  affordable  digital 
cameras  that  have  been  validated  for  use  with  the  DOCS  software,  the  DOCS  one-year  regulatory 
pilot  study  has  included  a  series  of  field  tests  in  which  the  following  commercially  available 
digital  cameras  will  be  evaluated  for  measuring  visible  opacity:  1)  Sony  Cybershot  DSC-WI,  2) 
Fuji  Finepix  E500,  3)  Nikon  Coolpix  5200  and  4)  Kodak  Model  6490. 

Another  important  lesson  learned  is  that  weather  conditions  must  be  considered  prior  to  use  of 
the  DOCS  as  well  as  Method  9.  Although  Method  9  does  not  specify  weather  conditions  under 
which  visible  opacity  should  not  be  estimated,  the  current  field  demonstration  results  confirmed 
that  the  performance  of  both  methods  diminishes  when  applied  during  adverse  weather 
conditions. 
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Finally,  a  recurring  statement  made  to  the  DOCS  field  demonstration  team  by  both  DoD  and 
regulatory  agency  personnel  was  that  no  alternative  opacity  measurement  technology  would  be 
considered  if  it  required  more  technical  effort  and/or  financial  resources  to  deploy  than  Method 
9.  Once  they  had  an  opportunity  to  use  the  technology,  participating  DoD  and  regulatory  agency 
personnel  found  that  the  DOCS’  ability  to  archive  digital  photographs  of  visible  opacity  as  well 
as  its  ease  of  use  in  estimating  opacity  were  highly  favorable  characteristics.  The  positive 
feedback  received  from  DoD  facilities  and  state  regulatory  agencies  illustrated  that  there  exists  a 
sizable  market  for  the  technology  once  regulatory  approval  is  obtained. 


6.5  End-User  Issues 

The  road  ahead  for  digitally  imaging  and  quantifying  plume  and/or  dust  opacity  has  been  given 
positive  initial  definition  by  the  DOCS.  However,  important  technology  maintenance  and 
regulatory  issues  remain.  These  are  summarized  in  the  following  bullets: 

•  New,  open-source  algorithms  must  be  developed.  Air  Force-funded  efforts  are  underway  to 
scope  these  possibilities  and  detennine  what  options  are  most  promising.  Funding  to  assure 
complete  development  and  technical  capability  remains  an  unknown. 

•  Long-term  support  for  a  DOCS  based  system  [i.e.,  help  desk  functions,  continued  technical 
development  support,  etc.]  would  have  to  be  agreed  to  and  supported  by  the  EPA  in 
conjunction  with  other  government  partners.  EPA  will  list  as  a  conditional  method  the 
performance-based  requirements  of  the  DOCS. 

•  The  EPA  will  pursue  permit-based  options  for  the  use  of  DOCS.  In  a  pennit  agreement  the 
DOCS  can  be  used  as  a  technique  for  assuring  that  the  air  pollution  control  system  is  being 
operated  properly  to  minimize  visible  emissions. 

6.  6  Approach  to  Regulatory  Compliance  and  Acceptance 

Since  the  ultimate  goal  of  the  current  field  demonstration  was  to  receive  regulatory  approval  for 
the  use  of  the  DOCS  by  regulated  DoD  facilities,  the  US  Environmental  Protection  Agency 
(EPA)  as  well  as  several  state  regulatory  agencies  were  involved  in  all  aspects  of  the  DOCS  field 
demonstration  study. 

The  US  EPA’s  Emissions  Measurement  Center  (EMC)  was  an  active  participant  in  the  field 
demonstration  program  and  was  integral  in  assessing  both  the  scientific  underpinnings  of  the 
DOCS  technology  as  well  as  the  regulatory  hurdles  associated  with  the  eventual  application  of  a 
digital-camera-based  opacity  technology  by  the  regulated  community.  The  DOCS  field  data 
demonstration  plan,  quality  assurance/quality  control  methodologies,  and  data  compilation  and 
interpretation  approaches  were  evaluated  and  endorsed  by  the  US  Environmental  Protection 
Agency’s  Emission  Measurement  Center  (EMC)  and  Method  9  experts  from  EPA  Region  VI 
(Dallas,  Texas)  and  EPA  Region  VIII  (Denver,  Colorado)  prior  to  collection  of  any  field  data. 

In  addition,  the  State  of  Alaska  Division  of  Air  Quality,  Anchorage,  Alaska;  State  of  Texas  Air 
Quality  Division,  Austin,  Texas;  and  the  State  of  Utah  Division  of  Air  Quality,  Salt  Lake  City, 
Utah  provided  regulatory  oversight  for  the  DOCS  field  demonstrations  as  well  as  valuable 
insight  into  the  current  performance  limitations  associated  with  of  the  application  of  EPA 
Reference  Method  9. 
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APPENDIX  A:  Analytical  Methods  Supporting  the  Experimental  Design 


The  present  field  demonstration  study  is  focused  on  the  use  of  the  Digital  Opacity  Compliance 
System  or  DOCS.  The  DOCS,  which  is  an  innovative  technology  that  employs  digital  imaging 
technology  for  quantifying  visible  opacity,  has  been  developed  and  field  tested  as  a  technically 
defensible  and  economically  competitive  alternative  to  Method  9.  The  DOCS  uses  a 
commercial-off-the-shelf  (COTS)  digital  camera  to  capture  images  of  visible  opacity,  which  are 
then  downloaded  to  a  standard  personal  computer  and  analyzed  using  statistical  computer 
software.  Therefore,  there  are  no  unique  analytical  methods  for  the  technology  user  beyond 
taking  the  digital  photographs. 

In  the  field  application  of  the  DOCS  technology,  digital  photographs  of  visible  emissions  were 
taken  from  valid  positions  according  to  codified  Method  9  specifications.  Once  downloaded  to  a 
laptop  or  desktop  computer  on  which  the  DOCS  computer  software  had  been  installed,  the  digital 
images  were  evaluated  for  visible  opacity.  The  initial  steps  in  analyzing  the  digital  image  for 
opacity  include  1)  activating  the  DOCS  opacity  computer  program,  2)  retrieving  those  digital 
photographs  that  are  to  be  evaluated  for  visible  opacity  and  3)  using  the  computer  program  to 
draw  an  analysis  box  (or  grid)  around  that  portion  of  the  visible  emissions  that  will  be  analyzed 
(Figure  A-l). 


Figure  A-l  Application  of  the  DOCS  Software 
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After  the  computer  software  selects  the  purity  of  color  (i.e.,  saturation)  that  best  corresponds  to 
the  background,  the  opacity  of  the  image  is  calculated  based  on  the  pixels  contained  in  the 
analysis  box.  The  size  and  shape  of  the  analysis  box,  which  is  controlled  by  the  user  of  the 
software,  must  be  chosen  judiciously  because  the  final  opacity  measurement  will  ultimately 
depend  on  what  part  of  the  image  the  DOCS  software  identifies  as  background. 

To  ensure  statistical  validity  in  the  opacity  estimates,  an  eight-member  panel  consisting  of 
federal  government  civilian  personnel,  US  military  personnel  and  federal  government  contractors 
estimated  the  opacity  of  each  smoke  plume  captured  as  a  digital  image  by  using  the  DOCS 
computer  software.  Each  panel  member  was  provided  a  compact  disc  that  contained  all  of  the 
digital  photographs  taken  from  the  respective  smoke  school  as  well  as  the  DOCS  computer 
software  and  user  guide.  The  panel  members  were  required  to  work  independently  to  estimate 
the  plume  opacity  of  each  digital  photograph  using  the  furnished  computer  software.  Once  panel 
members  had  completed  their  analyses,  the  opacity  results  were  transferred  and  stored 
electronically  in  a  relational  database  for  subsequent  statistical  evaluation.  An  independent 
quality  control  officer  was  assigned  the  responsibility  of  maintaining  the  integrity  of  all  opacity 
data  including  the  opacity  results  generated  from  the  EPA-certified  transmissometer,  against 
which  the  DOCS  and  human  observer  opacity  data  were  compared. 

Beyond  the  opacity  estimates,  the  weather  conditions  at  each  of  the  field  demonstration  sites 
were  monitored.  During  the  field  demonstrations,  on-site  field  personnel  recorded  the  values  of 
specific  climatic  parameters  including  1)  mean  air  temperature,  2)  average  wind  speed,  3) 
maximum  wind  speed,  4)  wind  direction,  5)  sky  conditions,  6)  relative  humidity,  7)  visibility,  8) 
barometric  pressure,  9)  precipitation,  10)  horizontal  sun  angle  and  11)  vertical  sun  angle. 
Methods  used  to  estimate  the  value  of  each  climatic  parameter  are  summarized  in  Table  A-l. 


Table  A-l 


Methods  Used  to  Estimate  Weather  Conditions 


Parameter 

Method 

Mean  temperature 

Standard  liquid  thermometer  (Eastern  Technical  Associates,  Inc.) 

Average  wind  speed 

Standard  anemometer  (Eastern  Technical  Associates,  Inc.) 

Maximum  wind  speed 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration  -  www.nws.noaa.gov) 

Wind  direction 

Standard  anemometer  (Eastern  Technical  Associates,  Inc.) 

Sky  conditions 

Visual  observation 

Relative  Humidity 

Sling  Psychrometer  (Eastern  Technical  Associates,  Inc.) 

Visibility 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration  -  www.nws.noaa.gov) 

Barometric  Pressure 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration  -  www.nws.noaa.gov) 

Precipitation 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration  -  www.nws.noaa.gov) 

Horizontal  Sun  Angle 

Magnetic  compass  (Eastern  Technical  Associates,  Inc.) 

Vertical  Sun  Angle 

Abney  level  (Eastern  Technical  Associates,  Inc.) 
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APPENDIX  B:  Data  Quality  Assurance/Quality  Control  Plan 

1.0  INTRODUCTION 

The  US  Environmental  Protection  Agency  (EPA)  quality  assurance  policy  requires  that  all 
projects  conducted  for  or  on  behalf  of  the  EPA  that  involve  the  generation  of  environmental  data 
be  planned  and  documented  in  an  Agency-approved  quality  assurance  project  plan  (QAPP).  The 
QAPP,  which  describes  how  quality  assurance  (QA)  and  quality  control  (QC)  activities  will  be 
implemented  during  the  life  cycle  of  a  program,  project  or  task,  must  be  approved  by  the  Agency 
prior  to  the  start  of  any  data  collection.  In  the  current  study,  the  ESTCP  demonstration  plan 
(demo  plan)  will  serve  the  same  function  as  the  QAPP  and  the  EPA  EMC  has  approved  its 
implementation. 

To  generate  environmental  data  of  the  appropriate  quality  for  decision-making,  a  sampling 
program  should  consist  of  three  distinct  phases  including  planning,  implementation  and 
assessment.  The  first  phase  of  the  sampling  program  is  characterized  by  the  development  of  the 
Data  Quality  Objectives  (DQOs)  using  the  DQO  process  or  similar  systematic  data  quality¬ 
planning  tool.  DQOs  provide  statements  about  the  expectations  and  data  quality  requirements  of 
the  decision-maker.  In  the  second  phase  of  the  program,  the  QAPP  translates  these  expectations 
and  data  quality  requirements  into  measurement  performance  specifications  required  to  satisfy 
the  decision-maker’s  needs.  In  the  third  phase,  the  Data  Quality  Assessment  (DQA)  process  is 
used  to  evaluate  the  data  and  to  draw  defensible  conclusions  from  the  data  set.  The  following 
sections  describe  the  basic  approach  for  applying  the  EPA  DQO  and  DQA  processes  to  evaluate 
the  equivalency  of  the  digital  opacity  compliance  system  (DOCS)  to  EPA  Reference  Method  9. 

2.0  DATA  QUALITY  OBJECTIVES  (DQO)  PROCESS 

The  DQO  process  is  a  strategic  planning  approach  based  on  the  scientific  method  that  is  used  to 
prepare  for  a  data  collection  activity.  It  provides  a  systematic  procedure  for  defining  the  criteria 
that  a  data  collection  design  should  satisfy  including  when  to  collect  samples,  where  to  collect 
samples,  the  tolerable  level  of  decision  errors  for  the  study,  and  how  many  samples  to  collect. 

By  using  the  DQO  Process,  the  decision-maker  ensures  that  the  type,  quantity  and  quality  of 
environmental  data  used  in  decision-making  will  be  appropriate. 

The  DQO  process  consists  of  seven  steps,  which  are  described  below.  The  output  from  each 
step  influences  the  choices  that  will  be  made  later  in  the  process.  Even  though  the  DQO  Process 
is  depicted  as  a  linear  sequence  of  steps,  in  practice,  it  is  iterative,  i.e.,  the  outputs  from  one  step 
may  lead  to  reconsideration  of  prior  steps.  The  following  sections  illustrate  the  application  of  the 
DQO  process  to  generate  sampling  performance  criteria  for  evaluating  the  potential  for  an 
opacity  measurement  technology  to  be  considered  as  a  viable  alternative  to  EPA  Reference 
Method  9. 

Step  1 :  State  the  Problem 

Step  2:  Identify  the  Decision 

Step  3:  Identify  the  Inputs  to  the  Decision 

Step  4:  Define  the  Study  Boundary 

Step  5:  Develop  a  Decision  Rule 
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Step  6:  Specify  Tolerable  Limits  on  Decision  Errors 

Step  7:  Optimize  the  Sampling  Design 

STEP  1 :  STATE  THE  PROBLEM 

The  purpose  of  this  step  is  to  define  the  problem  so  that  the  focus  of  the  study  will  be 
unambiguous. 

Federal  opacity  standards  for  various  industries  are  found  in  40  CFR  Part  60  ( Standards  of 
Performance  for  New  and  Modified  Stationary  Sources).  These  standards  require  the  use  of  EPA 
Reference  Method  9  for  the  determination  of  visible  emissions  by  trained  and  certified  human 
observers.  While  EPA  Reference  Method  9  has  an  extensive  history  of  successful  employment, 
its  reliance  on  human  observation  renders  it  vulnerable  to  inaccurate  and/or  biased  interpretation 
of  results. 

Because  of  concerns  regarding  the  reliability  of  opacity  estimates  obtained  by  employing  EPA 
Reference  Method  9,  there  is  a  clear  need  for  the  development  of  well  tested  and  objective 
opacity  measurement  alternatives.  Use  of  a  digital  opacity  photographic  evaluation  system  has 
the  potential  for  minimizing  both  the  costs  associated  with  evaluating  plume  opacity  and  the 
subjectivity  inherent  in  employing  human  observers  for  opacity  estimation.  The  scope  of  the 
present  study  has  been  formulated  to  statistically  evaluate  whether  the  digital  optical  compliance 
system  (DOCS)  is  as  accurate  and  reliable  as  EPA  Method  9  in  quantifying  visible  opacity. 


STEP  2:  IDENTIFY  THE  DECISION 

The  purpose  of  this  step  is  to  define  the  decision  statement  that  the  study  will  attempt  to  resolve. 

The  principal  question  that  this  study  will  attempt  to  resolve  is  formulated  as  follows:  “Is  the 
DOCS  as  accurate  and  reliable  in  estimating  plume  opacity  as  EPA  Reference  Method  9?”  The 
alternative  actions  that  could  result  from  resolution  of  the  principal  study  question  are  1)  DOCS 
is  as  accurate  and  reliable  in  estimating  plume  opacity  as  EPA  Reference  Method  9  and, 
therefore,  is  a  suitable  alternative  for  estimating  visible  emissions  or  2)  DOCS  is  not  as  accurate 
and  reliable  in  estimating  plume  opacity  as  EPA  Reference  Method  9  and,  therefore,  is  not  an 
equivalent  alternative. 


STEP  3:  IDENTIFY  THE  INPUTS  TO  THE  DECISION 

The  purpose  of  this  step  is  to  identify  the  informational  inputs  that  will  be  required  to  resolve  the 
decision  statement  and  determine  which  inputs  require  environmental  measurements. 

To  determine  whether  DOCS  is  capable  of  estimating  visible  emissions  as  accurately  and  reliably 
as  EPA  Reference  Method  9,  DOCS  will  be  employed  to  estimate  the  opacity  of  visible 
emissions  produced  from  a  smoke  generator  used  during  a  series  of  EPA  visible-emissions- 
certification  programs  (/.<?.,  smoke  schools).  The  smoke  generator  will  be  equipped  with  an  in¬ 
line  transmissometer  that  will  establish  the  “true”  or  reference  baseline  opacity  level  against 
which  DOCS  will  be  evaluated. 
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The  primary  sources  of  information  to  resolve  the  principal  study  question  will  consist  of  the 
opacity  readings  recorded  simultaneously  by  the  1)  smoke  generator  as  measured  by  an  in-line 
transmissometer  and  2)  digital  opacity  compliance  system  (DOCS).  The  difference  in  opacity 
measurements  recorded  by  the  in-line  transmissometer  positioned  in  the  stack  of  the  smoke 
generator  and  those  opacity  readings  recorded  by  DOCS  will  generate  the  type  of  data  required 
for  defensible  decision-making. 


STEP  4:  DEFINE  THE  STUDY  BOUNDARY 

The  purpose  of  this  step  is  to  define  the  spatial  and  temporal  boundaries  that  are  covered  by  the 
decision  statement. 

The  characteristics  that  define  the  population  of  interest  are  the  opacity  levels  for  white  and  black 
smoke  as  measured  by  the  1)  in-line  transmissometer  and  2)  digital  opacity  compliance  system 
(DOCS).  Spatially,  the  decision  resulting  from  resolving  the  principal  study  question  will  apply 
to  any  and  all  situations  in  which  EPA  Reference  Method  9  would  be  utilized  to  estimate  visible 
emissions. 

The  timeframe  over  which  opacity  data  will  be  collected  during  the  present  study  will  correspond 
to  the  duration  of  the  smoke  school  field  certification  activities.  Specific  opacity  data  will  be 
collected  when  the  smoke  school  contract  coordinator  indicates  that  the  plume  emission  is  stable 
and  that  the  opacity  level  should  be  estimated. 


STEP  5 :  DEVELOP  A  DECISION  RULE 

The  purpose  of  this  step  is  to  define  the  parameter  of  interest,  specify  the  action  level,  and 
integrate  previous  DQO  outputs  into  a  single  statemen  t  that  describes  a  logical  basis  for 
choosing  among  alternative  actions. 

The  parameter  of  interest  in  this  study  is  the  absolute  value  of  the  mean  difference  between  the 
in-line  transmissometer  opacity  readings  estimated  over  the  full  range  of  opacity  levels  and  the 
opacity  readings  recorded  by  the  digital  opacity  compliance  system  (DOCS).  The  in-line 
transmissometer  opacity  readings  will  serve  as  a  reference  baseline  and  subtraction  of  those  data 
from  the  opacity  readings  reported  by  the  digital  opacity  compliance  system  (DOCS)  will  be 
used  to  estimate  the  mean  difference  between  the  two  opacity  measurement  systems.  Since  EPA 
Reference  Method  9  permits  a  human  observer’s  estimate  of  visible  opacity  to  deviate  by  as 
much  as  7.5%  (on  average)  from  the  transmissometer  opacity  reading  and  still  qualify  for 
certification,  a  parameter  value  ( mean  difference)  of  7.5%  will  serve  as  the  action  level  for  the 
current  study. 

If  the  absolute  value  of  the  mean  difference  in  plume  opacity  measured  between  the  in-line 
transmissometer  and  the  digital  opacity  compliance  system  (DOCS)  is  statistically  less  than  or 
equal  to  7.5%  at  the  99%  confidence  level,  the  decision-maker  will  conclude  that  the  digital 
opacity  compliance  system  (DOCS)  is  equivalent  to  EPA  Reference  Method  9.  On  the  other 
hand,  if  the  absolute  value  of  the  mean  difference  in  estimated  plume  opacity  measured  between 
the  in-line  transmissometer  and  the  digital  opacity  compliance  system  (DOCS)  is  statistically 
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greater  than  7.5%  at  the  99%  confidence  level,  the  decision-maker  will  conclude  that  the  digital 
opacity  compliance  system  (DOCS)  is  not  equivalent  to  EPA  Reference  Method  9. 

At  present,  there  are  no  known  limitations  or  inherent  biases  associated  with  using  the  digital 
opacity  compliance  system  (DOCS)  to  estimate  visible  opacity.  Moreover,  the  current  sampling 
design,  which  specifies  the  use  of  randomly  chosen  opacity  levels  as  well  as  the  use  of  multiple 
cameras  and  certified  smoke  readers  positioned  at  various  locations  surrounding  the  stack,  should 
significantly  reduce  the  impact  of  any  inherent  bias  of  DOCS  on  conclusions  drawn  from  the 
final  test  results.  However,  to  fully  characterize  the  limitations  of  DOCS,  the  degree  of  bias  (or 
systematic  error)  associated  with  DOCS  will  be  evaluated  as  a  function  of  the  visible  opacity 
level  as  well  as  camera  position  relative  to  the  sun  and  stack.  By  estimating  the  difference 
between  the  average  DOCS  opacity  readings  and  the  true  opacity  (as  measured  by  an  EPA 
certified  transmissometer)  at  each  increment  of  opacity  (from  0  to  100%  opacity  for  both  black 
and  white  smoke)  and  for  each  camera  position,  the  identification  of  any  biases  associated  with 
the  application  of  DOCS  will  be  quantified.  If  the  level  of  bias  is  found  to  be  significant,  it  will 
be  fully  characterized  and  identified  as  an  inherent  limitation  of  the  system. 

The  precision  of  DOCS  will  be  reflected  in  the  variance  and  standard  deviation  estimated  from 
sampling  data  (i.e.,  differences  in  paired  opacity  readings).  By  collecting  hundreds  of  randomly 
selected  paired  opacity  measurements  for  each  site  location,  it  is  anticipated  that  the  effect  of 
random  errors  will  be  minimized  and,  therefore,  site-specific  data  will  be  characterized  by 
relatively  high  precision.  However,  in  developing  the  Phase  I  experimental  design,  it  was 
recognized  that  the  DOCS  performance  could  vary  significantly  depending  on  climatic 
conditions.  In  other  words,  temperature,  humidity  and/or  other  environmental  factors  could 
potentially  change  the  response  and  accuracy  of  DOCS  in  measuring  visible  opacity.  In  the 
present  study,  the  overall  accuracy  and  reproducibility  of  DOCS  will  be  determined  by 
comparing  the  precision  of  the  system  (as  reflected  by  the  variance  and  standard  deviation  of 
paired  opacity  readings)  at  each  of  the  physical  locations.  Significant  variation  in  the  level  of 
DOCS  precision  as  a  function  of  location  will  not  only  highlight  the  importance  of 
climatic/environmental  conditions  on  system  usability  but  would  also  suggest  another  potential 
limitation  of  the  technology. 

Finally,  in  comparing  the  ability  of  DOCS  to  accurately  measure  visible  opacity  relative  to  EPA 
Method-9-certified  smoke  readers,  no  DOCS  data  will  be  excluded  from  the  overall  analysis.  In 
other  words,  all  DOCS  data  will  be  included  in  the  statistical  comparison  with  EPA  Method-9- 
certified  smoke  readers  (i.e.,  those  human  observers  whose  overall  opacity  recordings  during 
certification  testing  were  within  +  7.5  %  of  the  true  opacity  level  and  had  no  single  opacity 
reading  greater  than 

+  15%  of  the  true  opacity  level).  The  concept  of  outliers  has  no  particular  relevance  in 
evaluating  the  DOCS  technology  since  the  recorded  visible  opacity  from  this  system  is  a  function 
of  both  the  DOCS  software  and  how  effectively  the  operator  interprets  and  captures  the  plume 
(i.e.,  how  the  control  volume  is  drawn)  within  the  digital  photograph.  Of  greater  significance, 
however,  is  the  need  to  establish  the  inherent  variability  associated  with  an  operator’s  use  of 
DOCS  to  estimate  visible  opacity.  By  employing  multiple  individuals  to  interpret  the  identical 
digital  photographs  with  the  DOCS  software,  it  is  anticipated  that  one  of  the  most  valuable 
results  from  the  current  study  will  be  a  reliable  estimate  of  the  inherent  variability  of  DOCS. 


50 


STEP  6: 


SPECIFY  LIMITS  ON  DECISION  ERRORS 


The  purpose  of  this  step  is  to  specify  the  decision-maker ’s  tolerable  limits  on  decision  errors, 
which  are  used  to  establish  performance  goals  for  the  data  collection  design. 

There  are  two  types  of  decision  errors  that  can  occur  based  on  the  decision-maker’s  interpretation 
of  sampling  data:  false  positive  and  false  negative  decision  errors.  In  the  comparison  of  paired 
opacity  measurements,  a  false  positive  decision  error  will  be  defined  to  occur  when  the  decision¬ 
maker  concludes  (based  on  field  measurements)  that  the  absolute  value  of  the  mean  difference  in 
visible  opacity  between  measurements  is  equal  to  or  less  than  7.5%  when,  in  fact,  it  is  greater. 
Conversely,  a  false  negative  decision  error  would  occur  when  the  decision-maker  concludes 
(based  on  field  measurements)  that  the  absolute  value  of  the  mean  difference  in  visible  opacity 
between  measurements  is  equal  to  or  less  than  7.5%  when,  in  fact,  it  is  greater. 

An  action  level  is  defined  as  the  value  of  the  parameter  of  interest  ( i.e .,  mean  difference )  that 
would  cause  the  decision-maker  to  choose  between  alternative  actions.  For  the  current  study, 
the  action  level  (AL)  will  be  established  as  7.5%.  In  other  words,  if  the  absolute  value  of  the 
mean  difference  in  visible  opacity  between  measurements  is  statistically  greater  than  7.5%,  the 
decision-maker  would  be  led  to  conclude  that  the  digital  opacity  compliance  system  (DOCS)  is 
not  equivalent  to  EPA  Method  9.  Alternatively,  if  the  absolute  value  of  the  mean  difference  in 
visible  opacity  between  measurements  is  statistically  equal  to  or  less  than  7.5%,  the  decision¬ 
maker  would  conclude  that  the  digital  opacity  compliance  system  (DOCS)  is  equivalent  to  EPA 
Reference  Method  9. 

Statistically,  a  false  positive  decision  error  is  referred  to  as  a  Type  I  error  and  its  occurrence  rate 
(or  probability)  is  normally  represented  by  the  letter  “a.”  For  example,  if  the  decision-maker 
desired  to  limit  the  risk  of  committing  a  false  positive  decision  error  to  one  (1)  percent,  “a” 
would  be  assigned  a  value  of  0.01.  Similarly,  a  false  negative  decision  error  is  referred  to  as  a 
Type  II  error  and  is  typically  represented  by  the  letter  “(3.”  The  tolerable  risk  (or  probability)  of 
committing  a  Type  II  error  is  also  established  by  the  decision-maker  ( e.g .,  assigning  [!  equal  to 
0.01  is  equivalent  to  establishing  a  false  negative  error  rate  of  1%). 

In  the  current  study,  the  decision  of  whether  the  digital  opacity  compliance  system  (DOCS)  is 
equivalent  to  EPA  Method  9  depends  on  whether  the  absolute  value  of  the  mean  difference 
between  opacity  readings  between  an  in-line  transmissometer  and  DOCS  can  be  shown  to  be 
equal  to  or  less  than  7.5%.  Since  the  direction  of  the  mean  difference  (i.e.,  positive  or  negative) 
is  not  relevant  in  making  a  decision,  the  tolerable  false  positive  and  false  negative  decision  error 
rates  (a  and  P)  should  be  identical.  In  the  present  study,  the  false  positive  and  false  negative 
decision  error  rates  both  will  be  assigned  a  value  of  0.01. 

In  addition  to  specifying  the  false  positive  and  false  negative  decision  error  rates,  the  decision¬ 
maker  is  required  to  specify  a  range  of  possible  parameter  values  near  the  action  level  where  the 
consequences  of  committing  false  negative  decision  errors  are  relatively  minor  (in  the  DQO 
process  this  is  called  the  gray  region).  In  statistics,  the  width  of  this  interval  is  called  the 
minimum  detectable  difference  and  is  expressed  as  the  Greek  letter  delta  (A).  Since  the  EPA 
Reference  Method  9  protocol  requires  visible  emission  observers  to  record  plume  opacities  in  5% 
intervals,  for  the  present  study,  A  will  be  arbitrarily  set  to  half  of  this  value,  i.e.,  2.5%. 
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STEP  7: 


OPTIMIZE  THE  SAMPLING  DESIGN 


The  purpose  of  this  step  is  to  identify  a  resource-effective  data  collection  design  for  generating 
data  that  are  expected  to  satisfy  the  DQOs. 

The  minimum  number  of  samples  (n)  necessary  to  satisfy  the  established  decision  error  rates  (a, 
P)  can  be  estimated  by  using  Equation  1.0. 


A2 


+  (0.5).Z2o 


(1.0) 


where 

n  =  minimum  number  of  samples  required  to  meet  rates  of  decision  errors 
cr  =  estimated  standard  deviation  (from  preliminary  studies) 
cr2  =  estimated  variance  (from  pilot  study) 

Z i-p  =  the  (1-P)  percentile  of  the  standard  normal  distribution  (from  standard  statistical 
tables) 

Zi_a  =  the  (1-a)  percentile  of  the  standard  normal  distribution  (from  standard  statistical 
tables) 

A  =  minimum  detectable  difference 

Short-tenn  visible  opacity  studies  conducted  at  Hill  AFB  using  the  digital  opacity  compliance 
system  (DOCS)  indicated  that,  at  the  30%  opacity  level,  this  measurement  system  has  an 
apparent  standard  deviation  of  3.44%  (cr).  Substituting  a  standard  deviation  (a)  of  3.44%,  a 
minimum  detectable  difference  (A)  of  2.5%,  and  tolerable  false  positive  (a)  and  false  negative 
(P)  decision  error  rates  of  0.01  into  Equation  1.0,  generates  forty-three  (43)  as  the  minimum 
number  of  samples  required  to  satisfy  the  quantitative  DQOs  (note  that  from  standard  statistical 
tables,  the  values  of  Z\_a  and  Z i_p  are  equal  to  2.3).  In  actuality,  since  the  digital  opacity 
compliance  system  (DOCS)  will  be  evaluated  against  EPA  Method  9  using  both  white  and  black 
smoke,  the  minimum  number  of  samples  required  to  satisfy  the  DQOs  should  be  twice  the 
estimated  amount  or  eighty-six  (86)  samples  (43  samples  of  both  black  and  white  smoke).  In 
practice,  the  minimum  number  of  samples  estimated  using  Equation  1.0  identifies  only  a  lower 
bound  for  the  required  number  of  samples.  To  ensure  that  the  data  set  will  be  of  a  sufficient 
size  to  draw  defensible  conclusions  (z'.e.,  decisions  supported  by  a  relatively  high  level  of 
confidence),  the  decision-maker  should  evaluate  the  cost  of  obtaining  substantially  more  samples 
than  the  minimum  number  estimated  from  using  Equation  1.0.  Once  the  data  have  been 
collected,  the  data  quality  assessment  (DQA)  process  should  be  applied  so  that  defensible 
conclusions  may  be  drawn  from  the  data  set. 


3.0  DATA  QUALITY  ASSESSMENT  (DQA)  PROCESS 

Data  quality  assessment  (DQA)  is  the  scientific  and  statistical  evaluation  of  data  to  detennine  if 
the  data  obtained  are  of  the  right  type,  quality  and  quantity  to  support  their  intended  use.  The 
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five  steps  of  the  USEPA  data  quality  assessment  (DQA)  process  include  1)  review  the  data 
quality  objectives  and  sampling  design,  2)  conduct  a  preliminary  data  review,  3)  select  the 
statistical  test,  4)  verify  the  assumptions  of  the  statistical  test  and  5)  draw  conclusions  from  the 
data. 

The  first  step  in  the  DQA  process  will  involve  a  review  of  the  DQO  outputs  and  sampling  design 
to  ensure  consistency.  Typical  activities  that  characterize  this  step  include  the  review  of  study 
objectives  and  assigning  the  tolerable  probability  limits  on  decision  errors.  During  the  second 
step  of  the  DQA  process,  quality  assurance  information  is  reviewed  to  ensure  that  the  data 
quality  conforms  to  the  decision-makers  requirements  and  to  identify  any  particular  patterns  or 
relationships. 

Following  the  completion  of  the  preliminary  data  review,  the  appropriate  statistical  procedure  for 
analyzing  the  experimental  data  is  selected  during  the  third  step  of  the  DQA  process.  For 
example,  in  the  present  study,  since  the  measurement  of  visible  opacity  recorded  by  the  in-stack 
transmissometer  will  be  compared  with  a  simultaneous  visibility  opacity  measurement  recorded 
by  the  DOCS,  evaluation  of  the  absolute  value  mean  of  the  paired  differences  will  be  employed 
as  a  measure  of  method  equivalency.  Specifically,  a  paired  /-test  will  be  used  to  assess  whether 
the  absolute  value  of  the  true  mean  of  the  paired  differences  (<5)  is  statistically  equal  to  or  less 
than  7.5%. 

In  the  fourth  step  of  the  DQA  process,  the  underlying  assumptions  of  the  statistical  test  are 
examined  to  determine  if  corrective  action,  which  may  include  the  possibility  of  collecting 
additional  data,  is  necessary.  Interpretation  of  the  experimental  data  results  and  the  drawing  of 
defensible  study  conclusions  characterize  the  fifth  step  of  the  DQA  process.  The  following 
section  provides  a  brief  discussion  of  how  the  paired  data  set  may  be  used  to  estimate  the 
absolute  value  of  the  mean  difference  in  opacity  readings  between  the  in-line  stack 
transmissometer  and  DOCS  data.  An  example  of  the  data  analysis  approach  is  provided  to 
illustrate  how  defensible  conclusions  may  be  drawn  from  the  data  set. 

4.0  STATISTICAL  PROCEDURES  FOR  EVALUATING  EQUIVALENCY  OF 
ALTERNATIVES  TO  EPA  REFERENCE  METHOD  9 

A  standard  statistical  procedure  for  comparing  the  equivalency  of  two  measurement  methods  is 
to  employ  significance  testing.  In  significance  testing,  a  null  hypothesis  (Ha)  is  developed  that 
will  be  assumed  to  be  true  in  the  absence  of  strong  quantitative  evidence  to  the  contrary.  The 
strength  of  the  data  may  be  evaluated  statistically  using  a  paired  sample  /-test.  The  results  of  the 
paired  sample  /-test  will  provide  the  basis  for  either  rejecting  or  not  rejecting  the  null  hypothesis, 
H0. 

The  null  hypothesis  (/!„)  for  the  present  study  may  be  stated  as  follows:  “the  absolute  value  of 
the  true  mean  difference  between  the  transmissometer  and  the  DOCS  opacity  measurement 
methods  is  greater  than  7.5%”  This  statement  reflects  the  decision-maker’s  initial  assumption 
that  the  two  opacity  measurement  methods  are  not  equivalent.  To  reach  a  determination  that  the 
DOCS  is  equivalent  to  EPA  Method  9,  the  strength  of  the  data  must  be  sufficient  to  reject  the 
null  hypothesis.  Similarly,  the  alternative  hypothesis  (Ha)  may  be  constructed  as  follows:  “ the 
absolute  value  of  the  true  mean  difference  between  the  transmissometer  and  the  DOCS  opacity 
measurement  method  is  equal  to  or  less  than  7.5% .”  If  the  strength  of  the  data  is  sufficient  to 


53 


reject  the  null  hypothesis,  the  decision-maker  will  conclude  that  the  alternative  hypothesis  (Ha)  is 
true  ( i.e .,  DOCS  is  equivalent  to  EPA  Method  9). 

In  statistical  terms,  these  hypotheses  can  be  presented  as  follows: 

Null  Hypothesis  (I/o):  8o> 7.5% 

Alternative  Hypothesis  (Ha):  So  <7.5% 


Where 

So=  true  mean  difference  (opacity  measured  using  DOCS  -  opacity  measured  using 

transmissometer) 

Since  the  true  mean  difference  between  the  two  visible  opacity  measurement  methods  (S0)  can 
never  be  known  exactly,  it  must  be  estimated  by  calculating  the  average  of  the  differences. 
Equation  1 . 1  may  be  used  to  calculate  the  average  of  the  paired  opacity  differences  from  the 
sampling  data. 


d 


i-n 

Z  d: 
i=l  1 
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1 

n 


Ik-d) 


(1.1) 


Where 


d  =  average  absolute  difference  between  paired  opacity  measurements 

Vi,„  V2,i  =  opacity  measurements  i  recorded  by  the  transmissometer  and  DOCS, 
respectively 

n  =  number  of  paired  measurements 

Equations  1.2  and  1.3  may  be  used  to  estimate  the  sample  variance  and  standard  error  of  the 
average  differences  between  opacity  readings,  respectively. 

,  -V 
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S 

d 
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Where 


di  =  difference  between  paired  opacity  measurements 
Sd  =  sample  variance 
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Sd  =  standard  error 

To  employ  the  paired  t -test  to  draw  defensible  conclusions  from  the  data  set  requires  that  the 
decision-maker  select  a  level  of  significance  (a)  from  which  a  critical  /-value  may  be  estimated. 
The  critical  /-value  is  compared  to  the  test  statistic  to  determine  if  the  strength  of  the  data  is 
sufficient  to  reject  the  null  hypothesis,  Ha.  Given  an  assigned  level  of  significance,  a,  and 
degrees  of  freedom  (n- 1),  the  following  expressions  define  the  critical  /-values  and  test-statistic 
(/test)  that  are  used  to  evaluate  whether  the  strength  of  the  data  is  sufficient  to  reject  the  null 
hypothesis. 


Critical /-value:  ta 


Where 


Test  statistic  :/t  t  = 


d  -  50 


sd 


So  =  absolute  value  of  mean  difference  (action  level  -  assumed  equal  to  7.5%) 

Once  the  critical  /-value  and  test  statistic  (/test)  have  been  estimated,  the  following  test  conditions 
are  employed  to  detennine  the  strength  of  the  field  data: 

Test  Condition:  If  /test  <  ta  ,  then  the  null  hypothesis,  H0,  is  rejected. 

r  n~l 

Test  Condition:  If  /test  >  ta  ,  the  data  do  not  support  rejection  of  the  null 

V  n~l 


hypothesis,  Ho. 

The  following  example  illustrates  the  use  of  these  statistical  principles  to  evaluate  the  ability  of 
DOCS  to  measure  plume  opacity  as  accurately  as  EPA  Method  9. 


EXAMPLE 


In  the  present  example,  it  is  assumed  that  the  decision-maker  has  applied  the  DQO  process  and 
has  determined  that  to  limit  the  false  positive  and  negative  decision  error  rates  to  5%,  a  minimum 
of  thirteen  (13)  samples  should  be  taken.  The  following  table  provides  a  summary  of  the  data 
from  the  13  paired  opacity  measurements  recorded  by  an  EPA  Method  9  in-stack 
transmissometer  and  DOCS.  Given  the  results  of  the  field  measurements,  the  decision-maker 
desires  to  know  whether  DOCS  is  capable  of  measuring  plume  opacity  as  accurately  as  EPA 
Method  9  at  the  5%  significance  level  (a  is  0.05).  In  other  words,  do  the  data  support  a  decision 
with  95%  confidence? 
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Measurement 

Number 

Transmissometer  Opacity 
Measurement  ( T) 

DOCS  Opacity 
Measurement  (D) 

Absolute  Value  of  the  Paired 
Differences:  T-D 

1 

15 

12.3 

2.7 

2 

50 

63.8 

13.8 

3 

30 

45.1 

15.1 

4 

45 

54.4 

9.4 

5 

65 

79.3 

14.3 

6 

90 

88.2 

1.8 

7 

5 

9.1 

4.1 

8 

25 

36.2 

11.2 

9 

20 

14.2 

5.8 

10 

40 

52.9 

12.9 

11 

55 

69.9 

14.9 

12 

95 

85.7 

9.3 

13 

35 

42.6 

7.6 

Statistical  Evaluation  Process  -  {Paired  t -test) 

Step  1.  Estimate  the  critical  /-value  from  standard  statistical  tables  using  a  0.05  level  of 

significance  and  12  {i.e.,  13  -  1)  degrees  of  freedom.  From  standard  tables,  the  following  critical 
t- value  was  found: 


t 


a 


2  ’ 


n  -1 


to. 025, 12  -  2.179 


Step  2.  Estimate  the  mean  of  the  absolute  value  of  the  paired  differences. 

cl  =  [(2.7)  +  (13.8)  +  (15.1)  +  ...  +  (7.6)]/13  =  9.45 


Step  3. 


2 

Estimate  the  variance  {sd  )  and  standard  deviation  (sci)  of  the  paired  differences 
using  Equation  1.2. 


9.45)2]/12 


Sd  -  — 


U\d\-d 


n  - 1 


[(2.7-9.45)2  +  (13.8-9.45)2  +  (15. 1-9.45)2  +  . . .  +  (7.6- 


=  22.38 


sd  =  V22.38  =  4.73 
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Step  4. 


Estimate  the  standard  error  of  the  paired  differences  using  Equation  E3. 


s 

d 


4.73 

Vn 


1.31 


Step  5.  Calculate  the  test  statistic  (ttest)  and  compare  it  to  the  critical  /-value  to  determine 

if  the  null  hypothesis  can  be  rejected  at  the  5%  significance  level. 

Test  statistic  :  ttest  =  =  9-45  '  7-5  =  1 .49 

sd  1.31 

yfn 

Since  ttest  <  t a  (i.e.,  1.49  <  2.179),  there  is  insufficient  evidence  to  reject  the  null 

— ,n- 1 
2 

hypothesis,  Ho. 

By  not  rejecting  the  null  hypothesis,  Ha,  the  decision-maker  would  conclude  that  the  absolute 
difference  between  values  measured  by  the  digital  opacity  compliance  system  (DOCS)  and  the 
transmissometer  opacity  measurement  system  is  greater  than  7.5%.  In  other  words,  the  decision 
would  be  that  DOCS  does  not  measure  plume  opacity  as  accurately  as  EPA  Reference  Method  9 
and  therefore,  the  two  measurement  systems  are  not  equivalent.  It  is  important  to  recognize  that 
whenever  the  null  hypothesis,  H0,  is  not  rejected,  the  power  of  the  significance  test  should  be 
evaluated  to  ensure  that  the  test  has  adequate  sensitivity  to  allow  the  decision-maker  to  reach  a 
defensible  conclusion.  An  equivalent  alternative  to  evaluating  the  power  of  the  significance  test 
is  to  calculate  the  required  sample  size  (m)  necessary  to  achieve  the  false  positive  and  negative 
decision  error  rates  and  compare  it  to  the  actual  number  of  samples  taken  (n).  The  required 
sample  size  (m)  necessary  to  satisfy  the  established  decision  error  rates  (a,  (!)  may  be  estimated 
using  Equation  1 .4. 


m  = 


+  0-5(O2 


(1.4) 


Test  Condition: 

If  m  <  n  The  decision-maker  would  conclude  that  a  sufficient  number  of  samples  have 
been  taken  to  determine  (with  95%  confidence)  that  the  true  mean  difference  is 
above  7.5%  and  therefore  DOCS  is  not  equivalent  to  EPA  Method  9  (acceptance 
of  the  null  hypothesis). 

If  m  >  n  The  decision-maker  would  conclude  that  an  insufficient  number  of  samples  have 
been  taken  to  support  a  decision  at  the  desired  level  of  confidence.  The  decision¬ 
maker  would  determine  that  the  data  suggest  that  the  true  mean  difference  is 
above  7.5%  and  that  DOCS  is  not  equivalent  to  EPA  Method  9.  However,  that 
decision  cannot  be  supported  at  the  95%  confidence  level. 
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Step  6.  Calculate  the  required  sample  size  (m)  necessary  to  satisfy  the  established 

decision  error  rates  (a,  P)  can  be  estimated  by  using  Equation  1 .4.  Note  that  Z\.a 
=  Z i_p=  1.65  when 
a  =  P  =  0.05. 

m  =  22-3ML65  +  L65)  +  0  5  .  65)2  =  32.6  (33  samples) 

(9.45 -7.5)2 


Since  m>n,  the  decision  maker  would  conclude  that  DOCS  is  not  statistically  equivalent  to  EPA 
Method  9  (i.e.,  acceptance  of  the  null  hypothesis)  but  that  an  insufficient  amount  of  data  was 
collected  to  support  that  decision  at  the  95%  confidence  level.  Alternatively,  the  decision-maker 
could  collect  additional  data  to  increase  the  level  of  confidence  in  the  decision. 
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APPENDIX  C  EPA  Reference  Method  9 


EMISSION  MEASUREMENT  TECHNICAL  INFORMATION  CENTER 
NSPS  TEST  METHOD 


Prepared  by  Emission  Measurement  Branch  EMTIC  TM-009 

Technical  Support  Division,  OAQPS,  EPA  October  25,  1990 

Method  9  -  Visual  Determination  of  the  Opacity  of  Emissions 

from  Stationary  Sources 


INTRODUCTION 

(a)  Many  stationary  sources  discharge  visible  emissions  into  the  atmosphere; 
these  emissions  are  usually  in  the  shape  of  a  plume.  This  method  involves  the 
determination  of  plume  opacity  by  qualified  observers.  The  method  includes 
procedures  for  the  training  and  certification  of  observers  and  procedures  to 
be  used  in  the  field  for  determination  of  plume  opacity. 

(b)  The  appearance  of  a  plume  as  viewed  by  an  observer  depends  upon  a  number 
of  variables,  some  of  which  may  be  controllable  in  the  field.  Variables  which 
can  be  controlled  to  an  extent  to  which  they  no  longer  exert  a  significant 
influence  upon  plume  appearance  include:  angle  of  the  observer  with  respect  to 
the  plume;  angle  of  the  observer  with  respect  to  the  sun;  point  of  observation 
of  attached  and  detached  steam  plume;  and  angle  of  the  observer  with  respect 
to  a  plume  emitted  from  a  rectangular  stack  with  a  large  length  to  width 
ratio.  The  method  includes  specific  criteria  applicable  to  these  variables. 

(c)  Other  variables  which  may  not  be  controllable  in  the  field  are 
luminescence  and  color  contrast  between  the  plume  and  the  background  against 
which  the  plume  is  viewed.  These  variables  exert  an  influence  upon  the 
appearance  of  a  plume  as  viewed  by  an  observer  and  can  affect  the  ability  of 
the  observer  to  assign  accurately  opacity  values  to  the  observed  plume. 

Studies  of  the  theory  of  plume  opacity  and  field  studies  have  demonstrated 
that  a  plume  is  most  visible  and  presents  the  greatest  apparent  opacity  when 
viewed  against  a  contrasting  background.  Accordingly,  the  opacity  of  a  plume 
viewed  under  conditions  where  a  contrasting  background  is  present  can  be 
assigned  with  the  greatest  degree  of  accuracy.  However,  the  potential  for  a 
positive  error  is  also  the  greatest  when  a  plume  is  viewed  under  such 
contrasting  conditions.  Under  conditions  presenting  a  less  contrasting 
background,  the  apparent  opacity  of  a  plume  is  less  and  approaches  zero  as  the 
color  and  luminescence  contrast  decrease  toward  zero.  As  a  result, 
significant  negative  bias  and  negative  errors  can  be  made  when  a  plume  is 
viewed  under  less  contrasting  conditions.  A  negative  bias  decreases  rather 
than  increases  the  possibility  that  a  plant  operator  will  be  incorrectly  cited 
for  a  violation  of  opacity  standards  as  a  result  of  observer  error. 

(d)  Studies  have  been  undertaken  to  determine  the  magnitude  of  positive 
errors  made  by  qualified  observers  while  reading  plumes  under  contrasting 
conditions  and  using  the  procedures  set  forth  in  this  method.  The  results  of 
these  studies  (field  trials)  which  involve  a  total  of  769  sets  of  25  readings 
each  are  as  follows: 

(1)  For  black  plumes  (133  sets  at  a  smoke  generator),  100  percent  of  the 
sets  were  read  with  a  positive  error  of  less  than  7.5  percent  opacity;  99 
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percent  were  read  with  a  positive  error  of  less  than  5  percent  opacity. 

(Note:  For  a  set,  positive  error  =  average  opacity  determined  by  observers' 

25  observations  -average  opacity  determined  from  transmissometer ' s  25 
recordings . ) 

(2)  For  white  plumes  (170  sets  at  a  smoke  generator,  168  sets  at  a  coal-fired 
power  plant,  298  sets  at  a  sulfuric  acid  plant) ,  99  percent  of  the  sets  were 
read  with  a  positive  error  of  less  than  7.5  percent  opacity;  95  percent  were 
read  with  a  positive  error  of  less  than  5  percent  opacity. 

(e)  The  positive  observational  error  associated  with  an  average  of  twenty- 
five  readings  is  therefore  established.  The  accuracy  of  the  method  must  be 
taken  into  account  when  determining  possible  violations  of  applicable  opacity 
standards . 

1.  PRINCIPLE  AND  APPLICABILITY 

1.1  Principle.  The  opacity  of  emissions  from  stationary  sources  is 
determined  visually  by  a  qualified  observer. 

1.2  Applicability.  This  method  is  applicable  for  the  determination  of  the 
opacity  of  emissions  from  stationary  sources  pursuant  to  §  60.11(b)  and  for 
visually  determining  opacity  of  emissions. 

2 .  PROCEDURES 

The  observer  qualified  in  accordance  with  Section  3  of  this  method  shall  use 
the  following  procedures  for  visually  determining  the  opacity  of  emissions. 

2.1  Position.  The  qualified  observer  shall  stand  at  a  distance  sufficient  to 
provide  a  clear  view  of  the  emissions  with  the  sun  oriented  in  the  140°  sector 
to  his  back.  Consistent  with  maintaining  the  above  requirement,  the  observer 
shall,  as  much  as  possible,  make  his  observations  from  a  position  such  that 
his  line  of  vision  is  approximately  perpendicular  to  the  plume  direction  and, 
when  observing  opacity  of  emissions  from  rectangular  outlets  (e.g.,  roof 
monitors,  open  baghouses,  noncircular  stacks),  approximately  perpendicular  to 
the  longer  axis  of  the  outlet.  The  observer's  line  of  sight  should  not 
include  more  than  one  plume  at  a  time  when  multiple  stacks  are  involved,  and 
in  any  case  the  observer  should  make  his  observations  with  his  line  of  sight 
perpendicular  to  the  longer  axis  of  such  a  set  of  multiple  stacks  (e.g.,  stub 
stacks  on  baghouses) . 

2.2  Field  Records.  The  observer  shall  record  the  name  of  the  plant,  emission 
location,  facility  type,  observer's  name  and  affiliation,  and  the  date  on  a 
field  data  sheet  (Figure  9-1) .  The  time,  estimated  distance  to  the  emission 
location,  approximate  wind  direction,  estimated  wind  speed,  description  of  the 
sky  condition  (presence  and  color  of  clouds),  and  plume  background  are 
recorded  on  a  field  data  sheet  at  the  time  opacity  readings  are  initiated  and 
completed . 
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Figure  9-1.  Record  of  visual  determination  of  opacity. 

Company 

Location 
Test  No. 

Date 

Type  Facility 
Control  Device 
Hours  of  Observation 
Observer 

Observer  Certification  Date  Observer  Affiliation 

Point  of  Emissions  Height  of  Discharge  Point 

CLOCK  TIME  Initial  Final 

OBSERVER  LOCATION 

Distance  to  discharge 

Direction  from  discharge 

Height  of  observation  point 

BACKGROUND  DESCRIPTION 

WEATHER  CONDITIONS 

Wind  Direction 

Wind  Speed 

Ambient  Temperature 

SKY  CONDITIONS  (clear, 
overcast,  %  clouds, 
etc . ) 

PLUME  DESCRIPTION 
Color 

Distance  Visible 
OTHER  INFORMATION 
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The  source  was/was  not  in  compliance  with  at  the  time  evaluation  was 

made . 


2.3  Observations.  Opacity  observations  shall  be  made  at  the  point  of 
greatest  opacity  in  that  portion  of  the  plume  where  condensed  water  vapor  is 
not  present.  The  observer  shall  not  look  continuously  at  the  plume  but 
instead  shall  observe  the  plume  momentarily  at  15-second  intervals. 

2.3.1  Attached  Steam  Plumes.  When  condensed  water  vapor  is  present  within 
the  plume  as  it  emerges  from  the  emission  outlet,  opacity  observations  shall 
be  made  beyond  the  point  in  the  plume  at  which  condensed  water  vapor  is  no 
longer  visible.  The  observer  shall  record  the  approximate  distance  from  the 
emission  outlet  to  the  point  in  the  plume  at  which  the  observations  are  made. 

2.3.2  Detached  Steam  Plume.  When  water  vapor  in  the  plume  condenses  and 
becomes  visible  at  a  distinct  distance  from  the  emission  outlet,  the  opacity 
of  emissions  should  be  evaluated  at  the  emission  outlet  prior  to  the 
condensation  of  water  vapor  and  the  formation  of  the  steam  plume. 

2.4  Recording  Observations.  Opacity  observations  shall  be  recorded  to  the 
nearest  5  percent  at  15-second  intervals  on  an  observational  record  sheet. 

(See  Figure  9-2  for  an  example.)  A  minimum  of  24  observations  shall  be 
recorded.  Each  momentary  observation  recorded  shall  be  deemed  to  represent  the 
average  opacity  of  emissions  for  a  15-second  period. 

2.5  Data  Reduction.  Opacity  shall  be  determined  as  an  average  of  24 
consecutive  observations  recorded  at  15-second  intervals.  Divide  the 
observations  recorded  on  the  record  sheet  into  sets  of  24  consecutive 
observations.  A  set  is  composed  of  any  24  consecutive  observations.  Sets 
need  not  be  consecutive  in  time  and  in  no  case  shall  two  sets  overlap.  For 
each  set  of  24  observations,  calculate  the  average  by  summing  the  opacity  of 
the  24  observations  and  dividing  this  sum  by  24.  If  an  applicable  standard 
specifies  an  averaging  time  requiring  more  than  24  observations,  calculate  the 
average  for  all  observations  made  during  the  specified  time  period.  Record 
the  average  opacity  on  a  record  sheet.  (See  Figure  9-1  for  an  example.) 


3.  QUALIFICATION  AND  TESTING 

3.1  Certification  Requirements.  To  receive  certification  as  a  qualified 
observer,  a  candidate  must  be  tested  and  demonstrate  the  ability  to  assign 
opacity  readings  in  5  percent  increments  to  25  different  black  plumes  and 

25  different  white  plumes,  with  an  error  not  to  exceed  15  percent  opacity  on 
any  one  reading  and  average  error  not  to  exceed  7.5  percent  opacity  in  each 
category.  Candidates  shall  be  tested  according  to  the  procedures  described  in 
Section  3.2.  Smoke  generators  used  pursuant  to  Section  3.2  shall  be  equipped 
with  a  smoke  meter  which  meets  the  requirements  of  Section  3.3.  The 
certification  shall  be  valid  for  a  period  of  6  months,  at  which  time  the 
qualification  procedure  must  be  repeated  by  any  observer  in  order  to  retain 
certification . 

3.2  Certification  Procedure.  The  certification  test  consists  of  showing  the 
candidate  a  complete  run  of  50  plumes--25  black  plumes  and  25  white  plumes-- 
generated  by  a  smoke  generator.  Plumes  within  each  set  of  25  black  and  25 
white  runs  shall  be  presented  in  random  order.  The  candidate  assigns  an 
opacity  value  to  each  plume  and  records  his  observation  on  a  suitable  form. 

At  the  completion  of  each  run  of  50  readings,  the  score  of  the  candidate  is 
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determined.  If  a  candidate  fails  to  qualify,  the  complete  run  of  50  readings 
must  be  repeated  in  any  retest.  The  smoke  test  may  be  administered  as  part  of 
a  smoke  school  or  training  program  and  may  be  preceded  by  training  or 
familiarization  runs  of  the  smoke  generator  during  which  candidates  are  shown 
black  and  white  plumes  of  known  opacity. 

3.3  Smoke  Generator  Specifications.  Any  smoke  generator  used  for  the 
purposes  of  Section  3.2  shall  be  equipped  with  a  smoke  meter  installed  to 
measure  opacity  across  the  diameter  of  the  smoke  generator  stack.  The  smoke 
meter  output  shall  display  in-stack  opacity  based  upon  a  pathlength  equal  to 
the  stack  exit  diameter,  on  a  full  0  to  100  percent  chart  recorder  scale.  The 
smoke  meter  optical  design  and  performance  shall  meet  the  specifications  shown 
in  Table  91.  The  smoke  meter  shall  be  calibrated  as  prescribed  in  Section 

3.3.1  prior  to  the  conduct  of  each  smoke  reading  test.  At  the  completion  of 
each  test,  the  zero  and  span  drift  shall  be  checked  and  if  the  drift  exceeds 
±1  percent  opacity,  the  condition  shall  be  corrected  prior  to  conducting  any 
subsequent  test  runs.  The  smoke  meter  shall  be  demonstrated,  at  the  time  of 
installation,  to  meet  the  specifications  listed  in  Table  9-1.  This 
demonstration  shall  be  repeated  following  any  subsequent  repair  or  replacement 
of  the  photocell  or  associated  electronic  circuitry  including  the  chart 
recorder  or  output  meter,  or  every  6  months,  whichever  occurs  first. 


TABLE  9-1  -  SMOKE  METER  DESIGN  AND  PERFORMANCE  SPECIFICATIONS 


Parameter 

Specification 

a . 

Light  Source 

Incandescent  lamp  operated  at  nominal 
rated  voltage 

b . 

Spectral  response  of  photocell 

Photopic  (daylight  spectral  response  of 
the  human  eye  -  Citation  3) 

c . 

Angle  of  view 

15B  maximum  total  angle 

d. 

Angle  of  projection 

15B  maximum  total  angle 

e . 

Calibration  error 

±3%  opacity,  maximum 

f . 

Zero  and  span  drift 

±1%  opacity,  30  minutes 

g- 

Response  time 

5  seconds 

3.3.1  Calibration.  The  smoke  meter  is  calibrated  after  allowing  a  minimum  of 
30  minutes  warm-up  by  alternately  producing  simulated  opacity  of  0  percent  and 
100  percent.  When  stable  response  at  0  percent  or  100  percent  is  noted,  the 
smoke  meter  is  adjusted  to  produce  an  output  of  0  percent  or  100  percent,  as 
appropriate.  This  calibration  shall  be  repeated  until  stable  0  percent  and 
100  percent  opacity  values  may  be  produced  by  alternately  switching  the  power 
to  the  light  source  on  and  off  while  the  smoke  generator  is  not  producing 
smoke . 

3.3.2  Smoke  Meter  Evaluation.  The  smoke  meter  design  and  performance  are  to 
be  evaluated  as  follows: 

3. 3. 2.1  Light  Source.  Verify  from  manufacturer's  data  and  from  voltage 
measurements  made  at  the  lamp,  as  installed,  that  the  lamp  is  operated  within 
±5  percent  of  the  nominal  rated  voltage. 
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3. 3. 2. 2  Spectral  Response  of  Photocell.  Verify  from  manufacturer's  data  that 
the  photocell  has  a  photopic  response;  i.e.,  the  spectral  sensitivity  of  the 
cell  shall  closely  approximate  the  standard  spectral-luminosity  in  (b)  of 
Table  91. 

3. 3. 2. 3  Angle  of  View.  Check  construction  geometry  to  ensure  that  the  total 

angle  of  view  of  the  smoke  plume,  as  seen  by  the  photocell,  does  not  exceed 
15°.  The  total  angle  of  view  may  be  calculated  from:  1=2  tan”1  (d/2 L) ,  where 

1  =  total  angle  of  view;  d  =  the  sum  of  the  photocell  diameter  +  the  diameter 

of  the  limiting  aperture;  and  L  =  the  distance  from  the  photocell  to  the 
limiting  aperture.  The  limiting  aperture  is  the  point  in  the  path  between  the 
photocell  and  the  smoke  plume  where  the  angle  of  view  is  most  restricted.  In 
smoke  generator  smoke  meters  this  is  normally  an  orifice  plate. 

3. 3. 2. 4  Angle  of  Projection.  Check  construction  geometry  to  ensure  that  the 

total  angle  of  projection  of  the  lamp  on  the  smoke  plume  does  not  exceed  15°. 
The  total  angle  of  projection  may  be  calculated  from:  1=2  tan”1  ( d/2L) ,  where 

1  =  total  angle  of  projection;  d  =  the  sum  of  the  length  of  the  lamp 

filament  +  the  diameter  of  the  limiting  aperture;  and  L  =  the  distance  from 
the  lamp  to  the  limiting  aperture. 

3. 3. 2. 5  Calibration  Error.  Using  neutral-density  filters  of  known  opacity, 
check  the  error  between  the  actual  response  and  the  theoretical  linear 
response  of  the  smoke  meter.  This  check  is  accomplished  by  first  calibrating 
the  smoke  meter  according  to  Section  3.3.1  and  then  inserting  a  series  of 
three  neutral-density  filters  of  nominal  opacity  of  20,  50,  and  75  percent  in 
the  smoke  meter  pathlength.  Filters  calibrated  within  2  percent  shall  be 
used.  Care  should  be  taken  when  inserting  the  filters  to  prevent  stray  light 
from  affecting  the  meter.  Make  a  total  of  five  nonconsecutive  readings  for 
each  filter.  The  maximum  error  on  any  one  reading  shall  be  3  percent  opacity. 

3. 3. 2. 6  Zero  and  Span  Drift.  Determine  the  zero  and  span  drift  by 
calibrating  and  operating  the  smoke  generator  in  a  normal  manner  over  a  1-hour 
period.  The  drift  is  measured  by  checking  the  zero  and  span  at  the  end  of 
this  period. 

3. 3. 2. 7  Response  Time.  Determine  the  response  time  by  producing  the  series 
of  five  simulated  0  percent  and  100  percent  opacity  values  and  observing  the 
time  required  to  reach  stable  response.  Opacity  values  of  0  percent  and  100 
percent  may  be  simulated  by  alternately  switching  the  power  to  the  light 
source  off  and  on  while  the  smoke  generator  is  not  operating. 


4 .  BIBLIOGRAPHY 

1.  Air  Pollution  Control  District  Rules  and  Regulations,  Los  Angeles  County 
Air  Pollution  Control  District,  Regulation  IV,  Prohibitions,  Rule  50. 

2.  Weisburd,  Melvin  I.,  Field  Operations  and  Enforcement  Manual  for  Air, 
U.S.  Environmental  Protection  Agency,  Research  Triangle  Park,  N.C., 
APTD-1100,  August  1972,  pp .  4.1-4.36. 

3.  Condon.  E.U.,  and  Odishaw,  H.,  Handbook  of  Physics,  McGraw-Hill  Co.,  New 
York,  N.Y.,  1958,  Table  3.1,  p.  6-52. 


64 


